Further to the other answers, which are indeed correct: no technical definition exists- barks $\iff$ dog, frankly- but 'modern' is a well defined concept outside of mathematics; and to a certain extent it is one to which the barkings of modern mathematics agree.
It was once the case that mathematicians believed proofs to uncover the neccessarily true- there must be numbers working thus, and those numbers must naturally have no zero divisors- there must be a geometry built thus, and this geometry must have angles in a triangle summing to $180^o$. With the exception of the work of Euclid (whose axioms were largely seen by others as immutable anyway), the theorems of mathematics were seen as universal truths, fouded in pure logic- facts about platonic ideals.
Except none of this was true.
Perhaps the first chink in the armour of this classical mathematics came with the work of Bolyai and Gauss, constructing consistent geometries where triangles behaved unusually (turning, as we all know, into modern Hyperbolic geometry), that seeped from a change in the hitherto 'immutable' axioms.
And from here the trickle began, which rushed and swelled with time, and burst the banks of mathematics as was: axioms became plastic, changeable at will, and with them the mathematics that followed from them. New concepts were created, and concepts of concepts, enriching and enlarging the mathematical landscape in ways that generations before could not have imagined.
Parallell to this explosion was the search for foundations for these axioms- the dying embers of platonism in the work of Frege, Russell and Whitehead; Hilbert's program, seeming at first promising, were spectacularly micturated on by Godel's incompleteness theorems. And it soon became (quite) clear, that any (provably) 'ultimate' description of mathematics was doomed to failure.
Modernism outside of mathematics is characterised by a certain relativism- an understanding that different perspectives can lead to different (equally valid) conclusions. In modern mathematics one has the reals and the p-adics, euclidean and non-euclidian geometries, topologies and metric spaces, groups, rings, algebras: sets and mereology- and we cannot claim one to be more valid than the others.
In modern mathematics, our truths are absolute but crucially contingent, the children of axioms in a pluralistic universe of possible postulates.
Of course some would say that 'modern' just means 'with categories', but that's not quite as neat- perhaps we can fit categories to 'post-modern' somehow....
