I assume you are looking at Samuel SPERRIN at No 36 St James Street in 1881. The numbers in this street changed in around 1908 and on the 1911 Census Samuel SPERRIN is listed at 9 Richmond Street but when he signs he gives his address as 16 St James Street which is on the corner of St James Street and Richmond Street.
from Ancestry 1911 Census Weston-super-Mare.
You might be able to check this by looking at Street Directories for early 20th Century.