PUBLIC has access to the database by default, but it can't access the data. You can REVOKE the PUBLIC:
REVOKE CONNECT ON DATABASE your_database FROM PUBLIC;
If you want this setting for all future databases, revoke CONNECT on the template1 database (default template database for creating a new database):
REVOKE CONNECT ON DATABASE template1 FROM PUBLIC;