No, this is not possible unless there were only 2 numbers to begin with or if there were an even number of numbers to begin with and the number which was removed was the median.
If there were only 2 numbers to begin with, the median is the same as the mean, so you can continue as before.
If there were an even number of numbers and you remove the median, we know that the median was in the list. By definition, the median was the average of the two numbers in the middle of the list, so the median must have been one of those two numbers (or else the median would be the average of two number which were either both greater than the median or both less than the median, depending on where the median was in the list). Then the other number in the middle was also the median, and removing one copy of the median leaves the same number of numbers to the left of the other median as to the right, so the new median is still the median.
To prove it is impossible in any other case, there are 2 cases:
Case 1: There were an odd number of numbers
It is sufficient to show that two lists with a given number of elements, a given median, and a given number to be removed can have different medians after removing the number. Consider the following list where $k$ and $l$ are variables:
median - $k$, median - $k$, $\dots$, median, median + $l$, median + $l$, $\dots$
If the number to be removed was the median, the new median is the average of median-$k$ and median + $l$, and you do not have any information on $k$ and $l$, so this is impossible.
If the number to be removed was greater than the median, choose $l$ so that median+$l$ is that number. The median is now the average of median and median-$k$, and you have no information about $k$, so this is impossible.
A similar analysis shows that it is impossible if the number to be removed was less than the median.
Case 2: There were an even number of numbers, and the number to be removed is not the median
Consider the following list of numbers:
median-$k$, median-$k$, $\dots$, median, median, median + $l$, median + $l$, $\dots$
Doing a similar analysis as to case 1 gives the result.