The configurations that is!
Some Notes Before Going to Town With Copy Pastes…
Please note that my setup for WeeWx, BelcherTown Weather Theme / Template, the web servers, and database are not a normal method of serving and storing data. Many configuration options listed below will need to be modified to suit your own installation.
This particular configuration below uses three machines, two being a virtual machine, and one being dedicated. The dedicated machine is for the SQL databases. The two virtual machines are for 1) the nginx reverse proxy / load balancer and 2) the weewx VM instance with mosquitto, weewx, nginx.
As always, like most other online posted guides, many configuration settings are of a non-normal or non-typical installations. Which may pose a security risk if exposed to the internet without proper configurations set.
Web Server
The web server and reverse proxy runs on nginx, both running on version 1.10.3. Both nginx servers (external and internal) listen on port 80, while the internal nginx server that hosts the actual weather weewx website solely listens on 80, and not 443 (SSL). The external nginx server (a basic load balancer in a manner of speaking), listens on port 443 and 80. Port 80 is always auto forwarded to 443 (SSL). Otherwise the WeeWx Belchertown Websockets will traditionally fail to function!
Database
The SQL server runs on MariaDB (a fork of mySQL), running version 10.3.14. The server is behind a software firewall, only permitting select IP addresses on LAN to port 3306. The mySQL server bind-address
is 0.0.0.0
so that it permits of outside traffic – providing a login method is given. The SQL server runs on its own hardware, split away from nginx and weewx machines.
Weather Station
Above all else, the weather station hardware. For any of this to work via a Davis weather station, you will need to obtain a WeatherLink USB Data Logger – it sells for 165 USD, but you can likely get it far cheaper from other online retailers, such as Amazon for 140 to 125 USD or Scientific Sales Inc for 105 to 115 USD, another site I’ve used is Scaled Instruments. They currently sell the 6510USB for 117 USD. Regardless, without this little piece of hardware, you’ll have a hard-pressed time getting data fed to WeeWx from a Davis Instruments console!
Additionally, there’s been developments from 3rd parties for WeeWx and RTL-SDR receivers/dongles. This would completely strip out the need for the weather console USB bridge. Granted, I’ve never attempted to use this extension on WeeWx, so I cannot vouch for data quality, stability, or functionality. If you’re wanting to dredge into uncharted waters, you can view that project here: https://github.com/matthewwall/weewx-sdr
WeeWx configuration
WeeWx for me has been modified a slight amount, in addition to many add ons / extensions installed and configured over the years. As of WeeWx 3.9.1, this is what my weewx.conf file looks like.
Currently, the following extensions are installed…
Extension Name Version Description cmon 0.16 Collect and display computer health indicators. Hfw 0.2.1 weewx support for plotting observational data using Highcharts. mqtt 0.19 Upload weather data to MQTT server. neowx 1.0 The most advanced weewx skin. Belchertown 0.9.1 A clean modern skin with real time streaming updates and interactive charts. Modeled after BelchertownWeather.com
cmon, and neowx are no longer proactively used. As cmon had stability issues for me, and neowx was in use before Belchertown skin.
NOTE: WeeWx loop data is also being dumped to the mySQL server in a second database. You’ll see it listed below as weewx_rapid
. You can find details about WeeWx’s Raw script here.
debug = 0 WEEWX_ROOT = /etc/weewx log_success = True log_failure = True socket_timeout = 20 version = 3.9.1 [Station] location = "Olathe Kansas" latitude = 38.809253 longitude = -94.781939 altitude = 1068, foot station_type = Vantage station_url = https://weewx.potatoforinter.net rain_year_start = 1 week_start = 6 [StdRESTful] [[StationRegistry]] register_this_station = false [[CWOP]] enable = true station = CWOPStationIDHere [[Wunderground]] enable = true station = WunderGroundStationIDHere password = ReplaceMeWithAPassword rapidfire = True [[MQTT]] server_url = mqtt://weewx:ReplaceMeWithAPassword@localhost:1883/ topic = weather unit_system = US binding = archive, loop aggregation = aggregate [Vantage] type = serial port = /dev/ttyUSB0 baudrate = 19200 iss_id = 1 timeout = 4 wait_before_retry = 1.2 max_tries = 4 model_type = 2 driver = weewx.drivers.vantage [StdReport] SKIN_ROOT = /etc/weewx/skins/ HTML_ROOT = /var/www/html/ data_binding = wx_binding report_timing = * * * * * log_success = True log_failure = True skin = Belchertown [[Highcharts_Belchertown]] HTML_ROOT = /var/www/html/ skin = Highcharts_Belchertown [[Belchertown]] HTML_ROOT = /var/www/html/ skin = Belchertown [[[Extras]]] belchertown_root_url = "https://weewx.potatoforinter.net" site_title = "Southern Olathe KS Weather" forecast_enabled = 0 earthquake_enabled = 0 twitter_enabled = 0 twitter_owner = bctrainers mqtt_websockets_enabled = 1 mqtt_websockets_host = "weewx.potatoforinter.net" mqtt_websockets_port = 2884 mqtt_websockets_ssl = 1 mqtt_websockets_topic = "weather/loop" disconnect_live_website_visitor = "9600000" [[Defaults]] [[[Units]]] [[[[Groups]]]] group_altitude = foot group_degree_day = degree_F_day group_pressure = inHg group_rain = inch group_rainrate = inch_per_hour group_speed = mile_per_hour group_speed2 = mile_per_hour2 group_temperature = degree_F [[[[StringFormats]]]] centibar = %.0f cm = %.2f cm_per_hour = %.2f degree_C = %.1f degree_F = %.1f degree_compass = %.0f foot = %.0f hPa = %.1f hour = %.1f inHg = %.3f inch = %.2f inch_per_hour = %.2f km_per_hour = %.0f km_per_hour2 = %.1f knot = %.0f knot2 = %.1f mbar = %.1f meter = %.0f meter_per_second = %.1f meter_per_second2 = %.1f mile_per_hour = %.0f mile_per_hour2 = %.1f mm = %.1f mmHg = %.1f mm_per_hour = %.1f percent = %.0f second = %.0f uv_index = %.1f volt = %.1f watt_per_meter_squared = %.0f NONE = " N/A" [[[[Labels]]]] day = " day", " days" hour = " hour", " hours" minute = " minute", " minutes" second = " second", " seconds" NONE = "" [[[[TimeFormats]]]] hour = %H:%M day = %X week = %X (%A) month = %x %X year = %x %X rainyear = %x %X current = %x %X ephem_day = %X ephem_year = %x %X [[[[Ordinates]]]] directions = N, NNE, NE, ENE, E, ESE, SE, SSE, S, SSW, SW, WSW, W, WNW, NW, NNW, N/A [[[[[DegreeDays]]]]] heating_base = 65, degree_F cooling_base = 65, degree_F [[[[[Trend]]]]] time_delta = 10800 time_grace = 300 [[[Labels]]] hemispheres = N, S, E, W latlon_formats = %02d, %03d, %05.2f [[[[Generic]]]] barometer = Barometer dewpoint = Dew Point ET = ET heatindex = Heat Index inHumidity = Inside Humidity inTemp = Inside Temperature outHumidity = Humidity outTemp = Outside Temperature radiation = Radiation rain = Rain rainRate = Rain Rate UV = UV Index windDir = Wind Direction windGust = Gust Speed windGustDir = Gust Direction windSpeed = Wind Speed windchill = Wind Chill windgustvec = Gust Vector windvec = Wind Vector extraTemp1 = Temperature1 extraTemp2 = Temperature2 extraTemp3 = Temperature3 rxCheckPercent = Signal Quality txBatteryStatus = Transmitter Battery windBatteryStatus = Wind Battery rainBatteryStatus = Rain Battery outTempBatteryStatus = Outside Temperature Battery inTempBatteryStatus = Inside Temperature Battery consBatteryVoltage = Console Battery heatingVoltage = Heating Battery supplyVoltage = Supply Voltage referenceVoltage = Reference Voltage [[[Almanac]]] moon_phases = New, Waxing crescent, First quarter, Waxing gibbous, Full, Waning gibbous, Last quarter, Waning crescent [StdConvert] target_unit = US [StdCalibrate] [[Corrections]] #foo = foo + 0.2 [StdQC] [[MinMax]] barometer = 26, 32.5, inHg pressure = 24, 34.5, inHg outTemp = -40, 120, degree_F inTemp = 10, 120, degree_F outHumidity = 0, 100 inHumidity = 0, 100 windSpeed = 0, 120, mile_per_hour rain = 0, 10, inch [StdWXCalculate] [[Calculations]] pressure = prefer_hardware barometer = prefer_hardware altimeter = prefer_hardware windchill = prefer_hardware heatindex = prefer_hardware dewpoint = prefer_hardware inDewpoint = prefer_hardware rainRate = prefer_hardware [StdTimeSynch] clock_check = 14400 max_drift = 2 [Raw] data_binding = raw_binding data_limit = 0 [StdArchive] archive_interval = 60 archive_delay = 1 record_generation = hardware loop_hilo = True data_binding = wx_binding [DataBindings] [[wx_binding]] database = archive_mysql table_name = archive manager = weewx.wxmanager.WXDaySummaryManager schema = schemas.wview.schema [[raw_binding]] database = raw_mysql table_name = raw manager = weewx.manager.Manager schema = user.raw.schema [Databases] [[archive_mysql]] database_type = mysql database_name = weewx [[raw_mysql]] database_type = mysql database_name = weewx_rapid [DatabaseTypes] [[mysql]] driver = weedb.mysql host = IP.IP.IP.IP user = ReplaceMeWithAUserName password = ReplaceMeWithAPassword [Engine] [[Services]] prep_services = weewx.engine.StdTimeSynch data_services = , process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate archive_services = weewx.engine.StdArchive, user.raw.RawService restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.mqtt.MQTT report_services = weewx.engine.StdPrint, weewx.engine.StdReport
As you can see above, I have weewx set to do the following…
- Report data to CWOP, send ‘rapidfire’ data to WUnderground, spam MQTT (WebSockets) with loop data – all the while saving to mySQL.
- Force an update once every minute to the webpage with the
report_timing
clause being* * * * *
, in the event websockets decide to go on a coffee break. - Using the Raw python script, dumping loop data to weewx_rapid.
- You can make extremely granular graphs with this sort of data!
- …At the expense of a massive database size.
- Using the same mySQL user : pass credentials for both databases.
You will need to change IP.IP.IP.IP
with your servers mySQL database IP address. Note this however: If your mySQL server is on the same machine as weewx, use localhost
, not 127.0.0.1
as the host. localhost is a socket, while 127.0.0.1 is TCP/IP. While miniscule, TCP/IP has overhead, while localhost is a socket connection.
You will need to change ReplaceMeWithAPassword
and ReplaceMeWithAUserName
in their respective locations.
So that’s WeeWx’s initial configuration out of the way… time for the nginx configuration!
As a reminder, I use nginx in two places. The front-end being a load-balancer of sorts – a reverse proxy, which delegates a hostname to an internal IP and sends the data back to the client. Please keep this in mind when you see the two configurations below.
The Reverse Proxy / Front-End configuration…
For WebSockets to function on the internet, and without losing my sanity, while passing WebSockets thru nginx to the mosquitto / mqtt daemon, open up port 2884 TCP on your firewall, and set its forwarded IP to the load balancer / reverse-proxy nginx server.
Additionally, there are two variables to set to forward requests to internally. One called $upstream
and the other called appserver
. These are both fairly easy to spot in the config. You’ll need to set both of these to the appropriate IP address inside your LAN, it’ll likely be the same IP address. The port on appserver
will need to remain however, otherwise WebSockets will fail.
The SSL cert uses Let’s Encrypt – it’s free, it works, it gets you a nifty padlock icon on your site to “show it’s secure”. For the LE SSL cert, it uses a wildcard + domain style. It makes most browsers happy. This will be covered after the nginx sections.
server { listen 80; server_name weewx.potatoforinter.net; return 301 https://$server_name$request_uri; } server { access_log /var/log/nginx/weewx.access.log; error_log /var/log/nginx/weewx.error.log; listen 443 ssl http2; server_name weewx.potatoforinter.net; underscores_in_headers on; include /etc/nginx/ssl.conf; set $upstream 192.168.20.203; location / { proxy_headers_hash_max_size 512; proxy_headers_hash_bucket_size 64; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; add_header Front-End-Https on; proxy_pass http://$upstream; proxy_buffering off; } } map $http_upgrade $connection_upgrade { default upgrade; '' close; } upstream appserver { server 192.168.20.203:9001; } server { server_name weewx.potatoforinter.net; listen 2884; ssl on; ssl_certificate /etc/letsencrypt/live/potatoforinter.net/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/potatoforinter.net/privkey.pem; location / { proxy_pass https://appserver; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade; proxy_read_timeout 86400; } }
The Backend (where weewx dumps static data to)…
Since this server doesn’t need any real vhost assigned, as it’s a stand-alone server, the following configuration is sufficient enough. This file is just a replacement to the file called default
within /etc/nginx/sites-available/
.
server { listen 80 default_server; root /var/www/html/; index index.php index.html; server_name _ weewx.potatoforinter.net; location / { try_files $uri $uri/ =404; } location ~ .php$ { include snippets/fastcgi-php.conf; fastcgi_pass unix:/var/run/php/php7.0-fpm.sock; } }
Certbot aka Let’s Encrypt and config’s…
So this is where things do get rather tricky, as mosquitto is hosted on another machine while the nginx reverse proxy is on another. You can’t exactly go creating multiple SSL certs for the same domain, that would be a bit absurd!
TODO: Automate SSL Certs to other *nix machines via SCP.
This command is what I use to generate a SSL cert: sudo certbot --server https://acme-v02.api.letsencrypt.org/directory -d *.potatoforinter.net,potatoforinter.net --manual --preferred-challenges dns-01 certonly -m ReplaceMe@WithAnEmailAddress.com
With this command, you’ll need to have your DNS editor at the ready. You’ll need to create a TEXT record. The text record will likely look something like this: _acme-challenge.potatoforinter.net. IN TXT "HcZGqj4UoNNIZzCBfmpQ48rsrerkJZdtV1uLjvqJC8-k4"
. The jumbled mass of text at the end is what CertBot will throw at you in the terminal to create as a TEXT record on your name servers.
For me, I host my DNS locally – so making changes are applied on the spot, with the TTL expiration dictating how often to refresh to a client.
To renew a CertBot/LE Cert, this command should do the trick: certbot renew --pre-hook "service nginx stop" --post-hook "service nginx start"
– it’ll kill nginx, check LE ssl certs for renewal, restart nginx.
Once your certs have been made, your certs will traditionally (on debian!) end up in a folder at /etc/letsencrypt/live/
with the final folder being the domain name. For me, it was potatoforinter.net
. Your SSL file within /etc/nginx/ssl.conf
should look like this…
ssl on; #Working SSL Dir: /etc/letsencrypt/live/potatoforinter.net/ ssl_certificate /etc/letsencrypt/live/potatoforinter.net/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/potatoforinter.net/privkey.pem; ssl_stapling on; ssl_stapling_verify on; ssl_session_cache shared:le_nginx_SSL:1m; ssl_ecdh_curve secp384r1; ssl_dhparam /etc/nginx/ssl/dhparam.pem; ssl_prefer_server_ciphers on; ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384; ssl_session_timeout 10m; ssl_session_cache shared:SSL:10m; ssl_session_tickets off; resolver 192.168.20.1 192.168.20.1 valid=300s; resolver_timeout 5s; add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload"; add_header Referrer-Policy no-referrer; add_header X-Content-Type-Options nosniff; add_header X-Frame-Options SAMEORIGIN; add_header X-XSS-Protection "1; mode=block"; add_header X-Robots-Tag none; add_header X-Download-Options noopen; add_header X-Permitted-Cross-Domain-Policies none; add_header X-Download-Options noopen; #add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://ssl.google-analytics.com; img-src 'self' https://ssl.google-analytics.com; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://themes.googleusercontent.com; frame-src https://potatoforinter.net; object-src 'none'";
That file is included within the server {}
calls, as shown above using include /etc/nginx/ssl.conf;
on the front-end reverse proxy.
I do not include the add_header Content-Security-Policy
to this ssl file, but keep it there for reference, primarily due to NextCloud and it’s stringent SSL requests and the joys of copy-pasta.
Why didn’t I put include ssl.conf
within the /etc/nginx/ssl/
folder? I dunno, didn’t feel like it?
Now, with all of that above set, it’s time for Mosquitto.
Mosquitto is a MQTT / WebSockets daemon. I found it to being the biggest PITA at first, as i didn’t quite understand how its configuration scheme functioned. I still don’t understand it to an extent, but i got it to work…. somehow.
Mosquitto will be installed on the weewx virtual machine, not the reverse proxy or database machine. It doesn’t use a lot of CPU time, nor does it use a lot of memory space. So it’s a perfect fit for the weewx virtual machine.
Assuming you’ve read this write up here:
https://obrienlabs.net/how-to-setup-your-own-mqtt-broker/ – the following should function as is.
pid_file /var/run/mosquitto.pid persistence false persistence_location /var/lib/mosquitto/ log_type all websockets_log_level 1023 connection_messages true log_dest file /var/log/mosquitto/mosquitto.log allow_anonymous true password_file /etc/mosquitto/passwd acl_file /etc/mosquitto/acl listener 1883 listener 8883 cafile /etc/mosquitto/certs/chain1.pem keyfile /etc/mosquitto/certs/privkey1.pem certfile /etc/mosquitto/certs/cert1.pem listener 9001 protocol websockets http_dir /var/www/html cafile /etc/mosquitto/certs/chain1.pem keyfile /etc/mosquitto/certs/privkey1.pem certfile /etc/mosquitto/certs/cert1.pem log_type error log_type warning log_type notice log_type information #include_dir /etc/mosquitto/conf.d
That is the config file that resides at /etc/mosquitto/mosquitto.conf
. I do not include the /etc/mosquitto/conf.d
in the config, and it is commented out – as there’s no need for it. In the config, this is what is going on…
- I am listening on port 1883, 8883 and a websockets port on 9001.
- Port 1883 is where weewx is dumping it’s data to.
- Port 8883 is a MQTT SSL port.
- Port 9001 is a WebSockets SSL port.
- I am logging everything, because this tool is a pain to work with when it decides to stop working for whatever reason.
So where do I get those certs from for mosquitto?
Easy as 1 2 3 actually…maybe! If you’ve not closed out nginx’s SSH terminal window for the reverse proxy / front end yet, there are three key files to yoink from there and to upload to the three given locations as seen above. For me, direct the SSH window to: /etc/letsencrypt/archive/
in there, you will have a folder made for the SSL applied domain. For me, it’s just potatoforinter.net, as weewx resides at https://weewx.potatoforinter.net/
If you’re not in the mood to download and upload files via SFTP (SSH+SCP) or SCP for that matter, you can use these commands to view, select, paste. You will need two SSH windows active, one for the reverse proxy server and the other for the weewx VM. In the reverse-proxy nginx web server, you”ll begin with…
- On the reverse-proxy nginx server,
cat chain1.pem
- Select the text, starting at the beginning of
-----BEGIN CERTIFICATE-----
and selecting to the end of-----END CERTIFICATE-----
- Left click the terminal window.
- Go to the other terminal window where mosquitto resides at.
cd /etc/mosquitto/certs/
nano chain1.pem
- Right click in the blank area to paste.
- CTRL O
- CTRL X
- Repeat the same for
privkey1.pem
andcert1.pem
. Replacingchain1.pem
from the example above to the two other respective file names.
Alternatively, it’d be much faster to just use FileZilla (or some sort of client that supports SFTP connections), logging in as root via sftp to both instances, navigating to both directories, downloading from one, and uploading to the other. Done.
Once all edits have been made…
Go ahead and restart all of the instances… that includes weewx, mosquitto, nginx on both machines.
[…] My WeeWx Setup […]