Sum[Floor[(a*i*(-3+2*i)+b)/m], {i, 1, n}] in closed algebraic form and output.

def Sum(a, b, m, n):
  return m * n * (n + 1) / 2 - (m - a) * n * (n + 1) * (2 * n + 1) / 6 + (m - a) * (m - 2 * a) * n * (n + 1) / 2 - (m - a) * (m - 2 * a) * (2 * m - 3 * a) * n / 6 - (m - a) * (m - 2 * a) * b / m + (m - a) * (m - 2 * a) * (2 * m - 3 * a) / 6