Sum[Floor[(a*i*(-3+2*i)+b)/m], {i, 1, n}] in closed algebraic form and output.

def sum(a, b, m, n):
    return sum(floor((a*i*(-3+2*i)+b)/m) for i in range(1, n+1))

sum(1,2,3,4)