Using configuration

Hi there,
I'm new to Splunk and the OPS team gave me the following files under the /opt/splunkforwarder/etc/system/local/:
$ ls -la /opt/splunkforwarder/etc/system/local/
total 16
drwx------ 2 root root  74 Jan  9 00:29 .
drwx------ 3 root root  19 Jan  8 22:29 ..
-rw------- 1 root root 171 Jan  8 22:29 inputs.conf
-rwx------ 1 root root 195 Jan  8 22:29 outputs.conf
-r-------- 1 root root 265 Jan  8 22:29 README
-rw------- 1 root root 375 Jan  8 22:29 server.conf

According to http://blogs.splunk.com/2015/08/24/collecting-docker-logs-and-stats-with-splunk/, I'd like to configure a single SplunkForwarder container to collect syslog for all the containers... I'd like also to disclose:

The company below is a BIG Splunk customer, but so far nobody from OPS team supports Docker
We are still on Splunk 6.2.x and so, we cannot use the Docker Native Driver http://blogs.splunk.com/2015/12/16/splunk-logging-driver-for-docker/

This is a host installation with the splunk-forwarder 6.2.x... The content of the files is as follows:
inputs.conf
[root@pe2enpmas300 npmo-server]# cat inputs.conf
[default]
host = pe2enpmas300.corp.company.net

[monitor:///var/log/messages]
disabled = false
sourcetype = syslog
_blacklist = \.(gz)$
index= sp-njsnginx-reference-e2eidx

outputs.conf
[root@pe2enpmas300 npmo-server]# cat outputs.conf
[tcpout]
defaultGroup = primary_indexers

[tcpout:primary_indexers]
server = oe2esstlg310.corp.company.net:9997, oe2esstlg311.corp.company.net:9997, oe2esstlg312.corp.company.net:9997
autoLB = true
```

server.conf
-----

```
[root@pe2enpmas300 npmo-server]# cat server.conf
[sslConfig]
sslKeysfilePassword = $1$Of8JPJZlRRS2

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = $1$brNdYNMjDka2
serverName = pe2enpmas300.corp.company.net
```

Questions
=====

* Can I just mount the settings in the Forwarder server?
* How can I debug this?

For instance, I just copied the files from the splunk forward from the host and I'm mounting in the data container... Let's start with the container in a separate docker-compose...

docker-compose-monitoring.yml
-----

Since I'm running docker as root (with SSL enabled), then I opened port `514`....

```yml
splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  environment:
    - SPLUNK_FORWARD_SERVER="oe2esstlg310.corp.company.net:9997,oe2esstlg311.corp.company.net:9997,oe2esstlg312.corp.company.net:9997"
  ports:
    - 514:514/udp
```

docker-compose.yml
-------

I just use extensions https://docs.docker.com/compose/extends/ and the ${HOSTNAME} variable substitution https://docs.docker.com/compose/compose-file/#variable-substitution

```
$ echo $HOSTNAME
pe2enpmas300.corp.company.net
```

For the configuration, I'm mounting it...

```
[root@pe2enpmas300 npmo-server]# ls -la monitor/splunk/
total 20
drwx------ 2 polkitd ssh_keys  95 Jan  9 01:00 .
drwx------ 3 root    root      19 Jan  9 01:40 ..
-rw------- 1 polkitd ssh_keys 171 Jan  8 22:29 inputs.conf
-rw------- 1 polkitd ssh_keys  45 Jan  9 01:00 migration.conf
-rwx------ 1 polkitd ssh_keys 195 Jan  8 22:29 outputs.conf
-r-------- 1 polkitd ssh_keys 265 Jan  9 00:51 README
-rw------- 1 polkitd ssh_keys 375 Jan  8 22:29 server.conf
```

Here's the file Dockerfile

```yml
splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local

splunkforwarder:
  extends:
    file: docker-compose-monitoring.yml
    service: splunkforwarder
  volumes_from:
    - "splunkforwarderData"

newww:
  build: roles/newww
  restart: always
  env_file: .env
  expose:
    - "5005"
  ports:
    - "80:8081"
  log_driver: "syslog"
  log_opt:
    syslog-tag: "newww"
    syslog-address: udp://${HOSTNAME}
```

Docker inspect command shows the mounted settings...

```
$ docker compose
    "Mounts": [
        {
            "Source": "/npmo-data/npmo-server/monitor/splunk",
            "Destination": "/opt/splunk/etc/system/local",
            "Mode": "rw",
            "RW": true
        }
    ],
```

I verified that the file is in the container as well...

```sh
root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash

root@bb3bb53aba78:/opt/splunk# ls -la
total 96
drwxr-xr-x  9 splunk splunk  4096 Jan  9 09:53 .
drwxr-xr-x  3 root   root      19 Oct 28 14:28 ..
drwxr-xr-x  3 splunk splunk  4096 Jun 26  2015 bin
-r--r--r--  1 splunk splunk    57 Jun 26  2015 copyright.txt
drwxr-xr-x 13 splunk splunk  4096 Jan  9 08:52 etc
drwxr-xr-x  2 splunk splunk    26 Jun 26  2015 include
drwxr-xr-x  4 splunk splunk  4096 Jun 26  2015 lib
-r--r--r--  1 splunk splunk 52503 Jun 26  2015 license-eula.txt
drwxr-xr-x  3 splunk splunk    55 Jun 26  2015 openssl
-r--r--r--  1 splunk splunk   842 Jun 26  2015 README-splunk.txt
drwxr-xr-x  3 splunk splunk    39 Jun 26  2015 share
-r--r--r--  1 splunk splunk 17634 Jun 26  2015 splunkforwarder-6.2.4-271043-linux-2.6-x86_64-manifest
drwxr-xr-x  6 splunk splunk    48 Jan  9 08:51 var

root@bb3bb53aba78:/opt/splunk# ls -la etc/system/local/
total 20
drwx------ 2 splunk splunk  95 Jan  9 09:00 .
drwxr-xr-x 7 splunk splunk  73 Jan  9 08:51 ..
-rw------- 1 splunk splunk 171 Jan  9 06:29 inputs.conf
-rw------- 1 splunk splunk  45 Jan  9 09:00 migration.conf
-rwx------ 1 splunk splunk 195 Jan  9 06:29 outputs.conf
-r-------- 1 splunk splunk 265 Jan  9 08:51 README
-rw------- 1 splunk splunk 375 Jan  9 06:29 server.conf
```

Any help is appreciated...

Looks like the config you received from Ops does not do what you thought it would do. See this snippet from your inputs.conf:
[monitor:///var/log/messages]

That stanza instructs a Splunk forwarder to tail a file on disk, not listen on a TCP/UDP port. As documented here, to listen on a UDP port, you'll need to do this:
[udp://514]

I suggest going back to your Ops folks to confirm the settings before baking them in.
To your questions:
If I understand what you are asking, yes you can mount a docker volume to effect configuration changes. Mount to the whole $SPLUNK_HOME/etc folder.
Troubleshooting steps. If you have access to Splunk, then you can search it for host=pe2enpmas300*, and failing that, host=pe2enpmas300* index=_internal. If the former sees nothing, you are not collecting data. If the latter sees nothing, then the forwarder is not forwarding data at all, not even the internal debug logs.
If you don't have Splunk access, fix that, because Splunk is cool. In the meantime, inspect the container's /var volume. /var/log/splunk/splunkd.log maps to what you should be seeing in that _internal line above.

Note that our best practice is to not use Splunk as a syslog server, even though it works fine. For production, we suggest setting up a syslog server (could be in a container...), and have a Splunk Forwarder ingest the syslog files on disk, so going back to using a line like the monitor://... stuff. This excellent blog post lays out the reasons why.

@marcellodesales at least one issue I see, please take a look on README.md

Splunk processes are running under splunk user.

We are using 1514 instead of standard 514 syslog port because ports below 1024 are reserved for root access only. See Run Splunk Enterprise as a different or non-root user.

I understand that you are running your docker daemon with root, but in the container I start splunk with splunk user. if you really want to use port 514 inside of the container - you should start splunk as root as well, for that you can specify environment variables SPLUNK_USER=root, SPLUNK_GROUP=root.
Also you have not enabled listening on port 514, see http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports
Another small issue - you do not need to specify SPLUNK_FORWARD_SERVER as you specify them on your own in outputs.conf. SPLUNK_FORWARD_SERVER exists only for people who does not have conf files and they want to configure it with this variable.
--
About the configuration files there are few options:

The right way will be to use Deployment Server, see http://docs.splunk.com/Documentation/Splunk/6.2.8/Updating/Aboutdeploymentserver So you will need only to specify deployment server and forwarder should pick up all the configurations from it. I should probably add a script which will allow to automatically set deployment server in the same way I do SPLUNK_FORWARD_SERVER

You can extend the image, build on top of my image: something like

FROM outcoldman/splunk:forwarder-6.2

COPY inputs.conf $SPLUNK_HOME/etc/system/local/
COPY outputs.conf $SPLUNK_HOME/etc/system/loca/
.... and so on ...

You can just mount to the $SPLUNK_HOME/etc/, don't worry about default settings - they will be copied over on startup.

--
Debugging
For debugging I would recommend to look inside of the splunkd.log, see
docker exec tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log

@halr9000 and @outcoldman, thank you so much for all the response here! I will go through today and I will place comments about our findings...

@halr9000
[udp://514]

This one of the reasons why it might not have worked... So, let me try that...

@outcoldman
SPLUNK_USER=root, SPLUNK_GROUP=root

I will be using the following on my ops-services-docker-compose.yml
splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  ports:
    - 514:514/udp
  environment:
    - SPLUNK_USER=root
    - SPLUNK_GROUP=root

Is that all I need?
I will monitor the splunk logs inside the container... I see that the container connected to the index servers:
$ docker exec -ti npmoserver_splunkforwarder_1 sh
[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@a59c0fdf35e9:/opt/splunk# tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log
01-11-2016 20:19:32.399 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splunkd_stdout.log'.
01-11-2016 20:19:32.400 +0000 INFO  WatchedFile - Will begin reading at offset=183 for file='/opt/splunk/var/log/splunk/splunkd_stderr.log'.
01-11-2016 20:19:32.412 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-11-2016 20:19:32.432 +0000 INFO  WatchedFile - Will begin reading at offset=955414 for file='/opt/splunk/var/log/splunk/metrics.log'.

I can see it connected to the hosts...
01-10-2016 14:56:43.229 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-10-2016 14:57:14.093 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-10-2016 14:57:43.309 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997

However, I see a few errors around SSL connectivity. Do I need to have the CERTS? or can I disable it?
01-11-2016 20:19:31.787 +0000 INFO  ServerConfig - My hostname is "a59c0fdf35e9".
01-11-2016 20:19:31.797 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-11-2016 20:19:31.797 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-11-2016 20:19:31.797 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-11-2016 20:19:31.797 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-11-2016 20:19:31.797 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-11-2016 20:19:31.801 +0000 INFO  LicenseMgr - Initing LicenseMgr

Thanks for any help!

Hi @halr9000  @outcoldman
Good news! The setup worked!!!

My question now is more related to the events... There's a mix of txt and json output... Is there a way to identify and differenciate among the events in syslog?
Thanks a lot

Nice! Should we close this issue now?

@outcoldman Actually I have to take it back... There was a splunkforwarder instance running on the host... :'( Here's what I have in the containers so far...
Docker-compose
Mounts the dir /var/log containing the /var/log/messages file. This HOST is writing the docker container logs to /var/log/messages...  This is volume-mounted in the docker container splunkforwarder.
splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local
    - /var/log:/var/log

splunkforwarder:
  image: outcoldman/splunk:6.2.4-forwarder
  restart: always
  environment:
    - SPLUNK_USER=root
    - SPLUNK_GROUP=root
  volumes_from:
    - "splunkforwarderData"

frontdoor:
  build: roles/registry
  restart: always
  env_file: .env
  ports:
    - "8080:8080"
  log_driver: "syslog"
  log_opt:
    syslog-tag: "frontdoor"

Logs
Can you confirm that this output below is good? I cannot see new messages going to splunk anymore :( Is there any other logs I should be looking at? I see in the bottom of the logs that the splunk forwarder is indeed connected to the 2 hosts...
root@f750b9a56059:/opt/splunk# ls -la /var/log/messages
-rw------- 1 root root 2842029 Jan 12 02:14 /var/log/messages
root@f750b9a56059:/opt/splunk# vim
bash: vim: command not found
root@f750b9a56059:/opt/splunk# vi $SPLUNK_HOME/var/log/splunk/splunkd.log
01-12-2016 02:14:18.906 +0000 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - My newly generated GUID is 0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - Found no site defined in server.conf
01-12-2016 02:14:18.907 +0000 INFO  ServerConfig - My hostname is "f750b9a56059".
01-12-2016 02:14:18.917 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFi
nal_ex:bad decrypt.
01-12-2016 02:14:18.917 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-12-2016 02:14:18.917 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-12-2016 02:14:18.917 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 02:14:18.917 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 02:14:18.928 +0000 INFO  LicenseMgr - Initing LicenseMgr
01-12-2016 02:14:18.928 +0000 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - connection_timeout=30
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - send_timeout=30
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - receive_timeout=30
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - squash_threshold=2000
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - strict_pool_quota=1
01-12-2016 02:14:18.929 +0000 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 02:14:18.929 +0000 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - closing stack mgr
01-12-2016 02:14:18.929 +0000 INFO  LMSlaveInfo - all slaves cleared
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 02:14:18.929 +0000 INFO  ServerRoles - Declared role=license_master.
01-12-2016 02:14:18.929 +0000 INFO  LMStackMgr - init completed [0E7035A5-7F3B-42ED-8735-7BEA3435BA74,Forwarder,runContext_splunkd=true]
01-12-2016 02:14:18.929 +0000 INFO  LicenseMgr - StackMgr init complete...
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - init'ing slaveId=0E7035A5-7F3B-42ED-8735-7BEA3435BA74 label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - enabling implicit feature set
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO  LMTracker - attempting to ping master=self from slave=0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.930 +0000 INFO  LMSlaveInfo - new slave='0E7035A5-7F3B-42ED-8735-7BEA3435BA74' created
01-12-2016 02:14:18.930 +0000 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Jan 12 00:00:00 2016), meaning that the master has a
lready rolled over. Ignore slave persisted usage.
"/opt/splunk/var/log/splunk/splunkd.log" 167 lines, 19388 characters
01-12-2016 02:14:19.128 +0000 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (No such fi
le or directory).  This is normal when splunk is first starting up.
01-12-2016 02:14:19.130 +0000 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 02:14:19.130 +0000 INFO  IndexProcessor - running splunkd specific init
01-12-2016 02:14:19.131 +0000 INFO  loader - Initializing from configuration
01-12-2016 02:14:19.132 +0000 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 02:14:19.134 +0000 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 02:14:19.134 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 02:14:19.282 +0000 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 02:14:19.282 +0000 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 02:14:19.287 +0000 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO  PipelineComponent - Launching the pipelines.
01-12-2016 02:14:19.288 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 02:14:19.288 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 02:14:19.293 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFi
nal_ex:bad decrypt.
01-12-2016 02:14:19.293 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 02:14:19.293 +0000 ERROR HTTPServer - SSL will not be enabled
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 02:14:19.341 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 02:14:19.342 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 02:14:19.342 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 02:14:19.342 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 02:14:19.342 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 02:14:19.371 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 02:14:49.131 +0000 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-12-2016 02:14:50.006 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 02:15:50.067 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 02:16:20.096 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 02:17:19.158 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997

@marcellodesales looks good for me. I see that you are monitoring /var/log/messages, also see that you could connect to the indexers. Try to less on var/log/messages and see that when you get new messages here you will get them in splunk as well.
I do not thing that it is good idea to mount /var/log into container in the same path, who knows what container can expect from this path (I do not run anything, which depends on it, but I cannot be 100% sure about that).

@outcoldman I changed the settings to the following...
splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local
    - /var/log/messages:/var/log/messages

That only loads the /var/log/messages... However, I still don't see anything in Splunk... The files under /opt/splunk/etc/system/local are loaded, correct? Because we see the dir being watched...
From inside the container, I see the logs coming to the /var/log/messages if I tail it... However, nothing is showing up in Splunk, although it says connected... :( Is there anything else I should be looking at?
root@be6fe6af3549:/opt/splunk# tail -f /var/log/messages
Jan 11 19:30:01 pe2enpmas300 systemd: Starting user-0.slice.
Jan 11 19:30:01 pe2enpmas300 systemd: Started Session 1186 of user root.
Jan 11 19:30:01 pe2enpmas300 systemd: Starting Session 1186 of user root.
Jan 11 19:30:01 pe2enpmas300 systemd: Removed slice user-0.slice.
Jan 11 19:30:01 pe2enpmas300 systemd: Stopping user-0.slice.
Jan 11 19:30:01 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:01.996Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:04 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:03.999Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:04 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:04.559Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:03:30:04 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:05 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:05.921Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:03:30:05 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:06 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:06.000Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:08 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:08.003Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:09 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:09.592Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:03:30:09 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:10 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:10.008Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:10 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:10.874Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:03:30:10 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:12 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:12.011Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:14 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:14.012Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:14 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:14.588Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:03:30:14 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:15 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:15.965Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:03:30:15 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:16 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:16.015Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:18 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:18.018Z: 0 queued changes, 0 open requests, 0 retries pending

The docker-compose logs show that the container tries to create the certs and initializes...
postgres_1            | taching: configured logging reader does not support reading
splunkforwarder_1     |
splunkforwarder_1     | This appears to be your first time running this version of Splunk.
splunkforwarder_1     |
splunkforwarder_1     | Splunk> Now with more code!
splunkforwarder_1     |
splunkforwarder_1     | Checking prerequisites...
nginx_1               | taching: configured logging reader does not support reading
splunkforwarder_1     | 	Checking mgmt port [8089]: New certs have been generated in '/opt/splunk/etc/auth'.
elasticsearch_1       | taching: configured logging reader does not support reading
splunkforwarder_1     | Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1     | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1     | Generating a 1024 bit RSA private key
splunkforwarder_1     | ...............................++++++
splunkforwarder_1     | ........................................++++++
auth_1                | taching: configured logging reader does not support reading
splunkforwarder_1     | writing new private key to 'privKeySecure.pem'
redis_1               | taching: configured logging reader does not support reading
splunkforwarder_1     | -----
splunkforwarder_1     | Signature ok
splunkforwarder_1     | subject=/CN=be6fe6af3549/O=SplunkUser
splunkforwarder_1     | Getting CA Private Key
splunkforwarder_1     | writing RSA key
splunkforwarder_1     | open
splunkforwarder_1     | 		Creating: /opt/splunk/var/lib/splunk
splunkforwarder_1     | 		Creating: /opt/splunk/var/run/splunk
splunkforwarder_1     | 		Creating: /opt/splunk/var/run/splunk/appserver/i18n
splunkforwarder_1     | 		Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
splunkforwarder_1     | 		Creating: /opt/splunk/var/run/splunk/upload
splunkforwarder_1     | 		Creating: /opt/splunk/var/spool/splunk
splunkforwarder_1     | 		Creating: /opt/splunk/var/spool/dirmoncache
splunkforwarder_1     | 		Creating: /opt/splunk/var/lib/splunk/authDb
splunkforwarder_1     | 		Creating: /opt/splunk/var/lib/splunk/hashDb
splunkforwarder_1     | 	Checking conf files for problems...
splunkforwarder_1     | 	Done
splunkforwarder_1     | All preliminary checks passed.
splunkforwarder_1     |
splunkforwarder_1     | Starting splunk server daemon (splunkd)...
splunkforwarder_1     | Done
splunkforwarder_1     |
splunkforwarder_1     | 2016-01-12 03:18:38.574 +0000 splunkd started (build 271043)

Then, I can see a few errors in the container
root@be6fe6af3549:/opt/splunk# grep -R "ERROR" var/log/splunk/splunkd.log
01-12-2016 03:18:38.572 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 03:18:38.572 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-12-2016 03:18:38.931 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 03:18:38.931 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 03:18:38.931 +0000 ERROR HTTPServer - SSL will not be enabled

Maybe this is the reason the container cannot forward the logs?

@marcellodesales sorry, I don't see anything unusual.
Just few recommendations:

Try the same setup without docker.
Check logs on indexers, if there are anything suspicious
Are you sure that blacklist works?
Are you sure that index exists?

@outcoldman

The host setup works, that's how I got the screenshot above...
Where can I find the logs on indexes?
As per 1, it works (from the host setup)
As per 1, it works (from the host setup)

The only delta I found was the SSL errors shown on the container... Here's the logs from the HOST without ANY ERROR...
[root@pe2enpmas300 npmo-server]# grep -R "ERROR" /opt/splunkforwarder/var/log/splunk/splunkd.log | wc -l
0

01-08-2016 20:01:27.070 -0800 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-08-2016 20:01:27.070 -0800 INFO  ServerConfig - My newly generated GUID is 6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.071 -0800 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-08-2016 20:01:27.071 -0800 INFO  ServerConfig - Found no site defined in server.conf
01-08-2016 20:01:27.071 -0800 INFO  ServerConfig - My hostname is "pe2enpmas300.corp.company.net".
01-08-2016 20:01:27.080 -0800 INFO  ServerConfig - Setting HTTP server compression state=on
01-08-2016 20:01:27.081 -0800 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-08-2016 20:01:27.081 -0800 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-08-2016 20:01:27.084 -0800 INFO  LicenseMgr - Initing LicenseMgr
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - connection_timeout=30
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - send_timeout=30
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - receive_timeout=30
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - squash_threshold=2000
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - strict_pool_quota=1
01-08-2016 20:01:27.084 -0800 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-08-2016 20:01:27.084 -0800 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-08-2016 20:01:27.084 -0800 INFO  LMStackMgr - closing stack mgr
01-08-2016 20:01:27.084 -0800 INFO  LMSlaveInfo - all slaves cleared
01-08-2016 20:01:27.140 -0800 INFO  LMConfig - created default pool=auto_generated_pool_forwarder for stack=forwarder
01-08-2016 20:01:27.140 -0800 INFO  LMStackMgr - added default pool=auto_generated_pool_forwarder for stack=forwarder
01-08-2016 20:01:27.142 -0800 INFO  LMConfig - created default pool=auto_generated_pool_free for stack=free
01-08-2016 20:01:27.142 -0800 INFO  LMStackMgr - added default pool=auto_generated_pool_free for stack=free
01-08-2016 20:01:27.142 -0800 INFO  ServerRoles - Declared role=license_master.
01-08-2016 20:01:27.142 -0800 INFO  LMStackMgr - init completed [6748789D-54FE-4389-B00D-F144248A0683,Forwarder,runContext_splunkd=true]
01-08-2016 20:01:27.142 -0800 INFO  LicenseMgr - StackMgr init complete...
01-08-2016 20:01:27.142 -0800 INFO  LMTracker - init'ing slaveId=6748789D-54FE-4389-B00D-F144248A0683 label=pe2enpmas300.corp.company.net [30,30,self]
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - enabling implicit feature set
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - attempting to ping master=self from slave=6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.143 -0800 INFO  LMSlaveInfo - new slave='6748789D-54FE-4389-B00D-F144248A0683' created
01-08-2016 20:01:27.143 -0800 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Fri Jan  8 00:00:00 2016), meaning that the master has a
lready rolled over. Ignore slave persisted usage.
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AdvancedXML state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=CustomRoles state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=DeployServer state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=GuestPass state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=KVStore state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=LDAPAuth state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=LocalSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=MultisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=NontableLookups state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=RollingWindowAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledReports state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=ScheduledSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SearchheadPooling state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - Setting feature=UnisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - setting masterGuid='6748789D-54FE-4389-B00D-F144248A0683'
01-08-2016 20:01:27.143 -0800 INFO  LMTracker - attempting to contact master=self from slave=6748789D-54FE-4389-B00D-F144248A0683 success
01-08-2016 20:01:27.143 -0800 INFO  LicenseMgr - Tracker init complete...
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: virtual address space size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: data segment size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: resident memory size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
01-08-2016 20:01:27.146 -0800 WARN  ulimit - Core file generation disabled
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: data file size: unlimited
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: open files: 4096 files
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: user processes: 63441 processes
01-08-2016 20:01:27.146 -0800 INFO  ulimit - Limit: cpu time: unlimited
01-08-2016 20:01:27.147 -0800 INFO  ulimit - Linux transparent hugetables support, enabled="always" defrag="always"
01-08-2016 20:01:27.150 -0800 INFO  loader - Splunkd starting (build 271043).
01-08-2016 20:01:27.150 -0800 INFO  loader - System info: Linux, pe2enpmas300.corp.company.net, 3.10.0-327.3.1.el7.x86_64, #1 SMP Fri Nov 20 05:40:26 EST 2015, x86_64.
01-08-2016 20:01:27.150 -0800 INFO  loader - Detected 4 (virtual) CPUs, 4 CPU cores, and 15880MB RAM
01-08-2016 20:01:27.150 -0800 INFO  loader - Maximum number of threads (approximate): 7940
01-08-2016 20:01:27.150 -0800 INFO  loader - Arguments are: "-p" "8089" "start"
01-08-2016 20:01:27.150 -0800 INFO  loader - Getting configuration data from: /opt/splunkforwarder/etc/myinstall/splunkd.xml
01-08-2016 20:01:27.151 -0800 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunkforwarder/etc/modules
01-08-2016 20:01:27.151 -0800 INFO  loader - loading modules from /opt/splunkforwarder/etc/modules
01-08-2016 20:01:27.152 -0800 INFO  loader - Writing out composite configuration file: /opt/splunkforwarder/var/run/splunk/composite.xml
01-08-2016 20:01:27.156 -0800 INFO  ServerRoles - Declared role=universal_forwarder.
01-08-2016 20:01:27.156 -0800 INFO  BundlesSetup - Setup stats for /opt/splunkforwarder/etc: wallclock_elapsed_msec=6, cpu_time_used=0.004996, shared_services_generation=
1, shared_services_population=1
01-08-2016 20:01:27.178 -0800 INFO  loader - Setting SSL configuration.
01-08-2016 20:01:27.178 -0800 INFO  loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
01-08-2016 20:01:27.178 -0800 INFO  loader - Using cipher suite TLSv1+HIGH:@STRENGTH
01-08-2016 20:01:27.178 -0800 INFO  loader - ECDH curve not configured
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 30 parameters: start_from, current_only, checkpointInter
val, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whiteli
st7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, suppress_text, renderXml
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 15 parameters: remoteAddress, process, user, addressFamily
, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interva
l, disabled, index
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled
, index, printSchema, baseline
01-08-2016 20:01:27.356 -0800 INFO  SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode,
samplingInterval, stats, disabled, index, showZeroValue, useEnglishOnly
01-08-2016 20:01:27.357 -0800 WARN  ClusteringMgr - Ignoring clustering configuration, the active license disables this feature.
01-08-2016 20:01:27.358 -0800 INFO  SHPoolingMgr - initing shpooling with: ht=60 rf=3 ct=60 st=60 rt=60 rct=5 rst=5 rrt=10 rmst=600 rmrt=600 pe=1 im=0 is=0 mor=5 pb=5 rep
_port= pptr=10
01-08-2016 20:01:27.358 -0800 INFO  SHPoolingMgr - shpooling disabled
01-08-2016 20:01:27.358 -0800 INFO  DS_DC_Common - Initializing the PubSub system.
01-08-2016 20:01:27.358 -0800 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
01-08-2016 20:01:27.366 -0800 INFO  DC:DeploymentClient - target-broker clause is missing.
01-08-2016 20:01:27.366 -0800 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-08-2016 20:01:27.366 -0800 INFO  DS_DC_Common - Deployment Client not initialized.
01-08-2016 20:01:27.366 -0800 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-08-2016 20:01:27.367 -0800 INFO  IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-08-2016 20:01:27.367 -0800 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-08-2016 20:01:27.367 -0800 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory).  This is normal when splunk is first starting up.
01-08-2016 20:01:27.369 -0800 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-08-2016 20:01:27.369 -0800 INFO  IndexProcessor - running splunkd specific init
01-08-2016 20:01:27.370 -0800 INFO  loader - Initializing from configuration
01-08-2016 20:01:27.372 -0800 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-08-2016 20:01:27.373 -0800 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-08-2016 20:01:27.373 -0800 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-08-2016 20:01:27.496 -0800 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-08-2016 20:01:27.496 -0800 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-08-2016 20:01:27.500 -0800 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO  PipelineComponent - Launching the pipelines.
01-08-2016 20:01:27.502 -0800 INFO  loader - Limiting REST HTTP server to 1365 sockets
01-08-2016 20:01:27.502 -0800 INFO  loader - Limiting REST HTTP server to 1365 threads
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - TailWatcher initializing...
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-08-2016 20:01:27.546 -0800 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
01-08-2016 20:01:27.547 -0800 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-08-2016 20:01:27.547 -0800 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-08-2016 20:01:27.600 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:01:57.371 -0800 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-08-2016 20:01:57.703 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:02:28.450 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:02:57.479 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:03:27.509 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:03:57.538 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:04:27.560 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:04:57.576 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:05:27.593 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:06:57.638 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-08-2016 20:07:27.652 -0800 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997

HOST: Works with the same configuration shared before...

CONTAINER: DOES NOT WORK so far...

Is there any Debug statement I can turn on in the container?

Hi @outcoldman
I got a Docker image based on CENTOS that works like the following:
We have a cached version of the rpm stored at http://sds-repo-int.qdc.company.com:8081/nexus/content/repositories/CTO.OPS-releases/com/intuit/CTO/OPS-releases/splunkforwarder/6.2.4/splunkforwarder-6.2.4-271043-lin ux-2.6-x86_64.rpm
Dockerfile
FROM richxsl/rhel7
MAINTAINER<EMAIL_ADDRESS>
ENV SPLUNK_HOME /opt/splunkforwarder
ENV SPLUNK_GROUP splunk
ENV SPLUNK_USER splunk
ENV SPLUNK_BACKUP_DEFAULT_ETC /var/opt/splunk

RUN groupadd -r ${SPLUNK_GROUP} \
    && useradd -r -m -g ${SPLUNK_GROUP} ${SPLUNK_USER}

COPY ./INTU-LATEST.repo /etc/yum.repos.d/

RUN yum remove -y subscription-manager
ENV LANG en_US.utf8

ADD http://sds-repo-int.qdc.company.com:8081/nexus/content/repositories/CTO.OPS-releases/com/intuit/CTO/OPS-releases/splunkforwarder/6.2.4/splunkforwarder-6.2.4-271043-lin
ux-2.6-x86_64.rpm .

RUN yum install -y splunkforwarder-6.2.4-271043-linux-2.6-x86_64.rpm && /opt/splunkforwarder/bin/splunk enable boot-start --accept-license --answer-yes && rm -f splunkfor
warder-6.2.4-271043-linux-2.6-x86_64.rpm

COPY entrypoint.sh /sbin/entrypoint.sh
RUN chmod +x /sbin/entrypoint.sh

WORKDIR /opt/splunkforwarder

# Configurations folder, var folder for everyting (indexes, logs, kvstore)
VOLUME [ "/opt/splunkforwarder/etc", "/opt/splunkforwarder/var" ]

ENTRYPOINT ["/sbin/entrypoint.sh"]

entrypoint.sh

Only for the forwarder running as non-daemon...

#!/bin/bash

set -e

${SPLUNK_HOME}/bin/splunk start --nodaemon --accept-license --answer-yes --no-prompt

Is there anything that this setup differs from yours? I noticed you do other things while setting up the server in entrypoint.sh.

Another false alarm... It is NOT working... Is there anything in the Docker level that I need to verify?
[deploy@pe2enpmas300 ~]$ sudo su
[root@pe2enpmas300 deploy]# cd /npmo-data/npmo-server/
[root@pe2enpmas300 npmo-server]# service splunk status
Splunk status:
splunkd is running (PID: 25381).
splunk helpers are running (PIDs: 25382).
[root@pe2enpmas300 npmo-server]# service splunk stop

Potential problems
Both docker images present this same problem: https://answers.splunk.com/answers/39497/unable-to-access-the-webui-and-im-getting-this-error-in-my-splunkd-log-anyone-know-how-to-fix-this.html
I'm not sure if this is caused by Docker images or if we need to share the server's cert.

@marcellodesales in your server.conf you have sslKeysfilePassword setup, do you actually map any your own certificates? Could you try to remove this line from conf?

@outcoldman Let me try that!

@outcoldman I get the same exact error I reported before and they are the same between the 2 docker images (yours and ours)...
docker logs
splunkforwarder_1 |
splunkforwarder_1 | Splunk> Now with more code!
splunkforwarder_1 |
splunkforwarder_1 | Checking prerequisites...
splunkforwarder_1 | 	Checking mgmt port [8089]: New certs have been generated in '/opt/splunkforwarder/etc/auth'.
splunkforwarder_1 | Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1 | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1 | Generating a 1024 bit RSA private key
splunkforwarder_1 | ..............................................................++++++
splunkforwarder_1 | ..........................++++++
splunkforwarder_1 | writing new private key to 'privKeySecure.pem'
splunkforwarder_1 | -----
splunkforwarder_1 | Signature ok
splunkforwarder_1 | subject=/CN=pe2enpmas300.corp.intuit.net/O=SplunkUser
splunkforwarder_1 | Getting CA Private Key
splunkforwarder_1 | writing RSA key
splunkforwarder_1 | 2016-01-12 15:20:40.571 +0000 splunkd started (build 271043)

splunkd.log
01-12-2016 15:20:40.922 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:20:40.922 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:20:40.927 +0000 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope rout
ines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 15:20:40.927 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 15:20:40.927 +0000 ERROR HTTPServer - SSL will not be enabled

@marcellodesales and please remove pass4SymmKey as well

@outcoldman ok Let me try it...

@outcoldman It finally solved the ERROR events... I'm recreating all the containers and verify...
[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@cd9d03dfce3b:/opt/splunk# vi var/log/
introspection/ splunk/
root@cd9d03dfce3b:/opt/splunk# vi var/log/splunk/splunkd.log
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My newly generated GUID is C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Found no site defined in server.conf
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My hostname is "cd9d03dfce3b".
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - connection_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - send_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - receive_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - squash_threshold=2000
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - strict_pool_quota=1
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - closing stack mgr
01-12-2016 15:45:34.604 +0000 INFO  LMSlaveInfo - all slaves cleared
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 15:45:34.604 +0000 INFO  ServerRoles - Declared role=license_master.
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - init completed [C3A35716-9824-4BDC-83D8-4C67FE8077DD,Forwarder,runContext_splunkd=true]
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - StackMgr init complete...
01-12-2016 15:45:34.604 +0000 INFO  LMTracker - init'ing slaveId=C3A35716-9824-4BDC-83D8-4C67FE8077DD label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - enabling implicit feature set
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - attempting to ping master=self from slave=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.605 +0000 INFO  LMSlaveInfo - new slave='C3A35716-9824-4BDC-83D8-4C67FE8077DD' created
01-12-2016 15:45:34.605 +0000 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Jan 12 00:00:00 2016), meaning that the mas
ter has already rolled over. Ignore slave persisted usage.
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
:1
01-12-2016 15:45:34.805 +0000 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
01-12-2016 15:45:34.813 +0000 INFO  DC:DeploymentClient - target-broker clause is missing.
01-12-2016 15:45:34.813 +0000 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Client not initialized.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-12-2016 15:45:34.814 +0000 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory).  This is normal when splunk is first starting up.
01-12-2016 15:45:34.815 +0000 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 15:45:34.815 +0000 INFO  IndexProcessor - running splunkd specific init
01-12-2016 15:45:34.816 +0000 INFO  loader - Initializing from configuration
01-12-2016 15:45:34.817 +0000 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 15:45:34.819 +0000 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 15:45:34.819 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 15:45:34.966 +0000 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 15:45:34.966 +0000 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Launching the pipelines.
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:45:35.031 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
:1
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My newly generated GUID is C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - Found no site defined in server.conf
01-12-2016 15:45:34.589 +0000 INFO  ServerConfig - My hostname is "cd9d03dfce3b".
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP server compression state=on
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 15:45:34.600 +0000 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - serverName=pe2enpmas300.corp.company.net guid=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - connection_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - send_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - receive_timeout=30
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - squash_threshold=2000
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - strict_pool_quota=1
01-12-2016 15:45:34.604 +0000 INFO  LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - closing stack mgr
01-12-2016 15:45:34.604 +0000 INFO  LMSlaveInfo - all slaves cleared
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 15:45:34.604 +0000 INFO  ServerRoles - Declared role=license_master.
01-12-2016 15:45:34.604 +0000 INFO  LMStackMgr - init completed [C3A35716-9824-4BDC-83D8-4C67FE8077DD,Forwarder,runContext_splunkd=true]
01-12-2016 15:45:34.604 +0000 INFO  LicenseMgr - StackMgr init complete...
01-12-2016 15:45:34.604 +0000 INFO  LMTracker - init'ing slaveId=C3A35716-9824-4BDC-83D8-4C67FE8077DD label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - enabling implicit feature set
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - attempting to ping master=self from slave=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.605 +0000 INFO  LMSlaveInfo - new slave='C3A35716-9824-4BDC-83D8-4C67FE8077DD' created
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedSearchCommands state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AdvancedXML state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=CustomRoles state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=GuestPass state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=KVStore state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=LDAPAuth state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=ScheduledAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LMTracker - Setting feature=UnisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO  LicenseMgr - Tracker init complete...
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: virtual address space size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: data segment size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: resident memory size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: core file size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: data file size: unlimited
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: open files: 1048576 files
01-12-2016 15:45:34.609 +0000 INFO  ulimit - Limit: user processes: 1048576 processes
01-12-2016 15:45:34.611 +0000 INFO  loader - Splunkd starting (build 271043).
01-12-2016 15:45:34.611 +0000 INFO  loader - System info: Linux, cd9d03dfce3b, 3.10.0-327.3.1.el7.x86_64, #1 SMP Fri Nov 20 05:40:26 EST 2015, x86_64.
01-12-2016 15:45:34.611 +0000 INFO  loader - Detected 4 (virtual) CPUs, 4 CPU cores, and 15880MB RAM
01-12-2016 15:45:34.611 +0000 INFO  loader - Maximum number of threads (approximate): 7940
01-12-2016 15:45:34.611 +0000 INFO  loader - Arguments are: "-p" "8089" "start"
01-12-2016 15:45:34.611 +0000 INFO  loader - Getting configuration data from: /opt/splunk/etc/myinstall/splunkd.xml
01-12-2016 15:45:34.612 +0000 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/etc/modules
01-12-2016 15:45:34.612 +0000 INFO  loader - loading modules from /opt/splunk/etc/modules
01-12-2016 15:45:34.613 +0000 INFO  loader - Writing out composite configuration file: /opt/splunk/var/run/splunk/composite.xml
01-12-2016 15:45:34.617 +0000 INFO  ServerRoles - Declared role=universal_forwarder.
01-12-2016 15:45:34.643 +0000 INFO  loader - Setting SSL configuration.
01-12-2016 15:45:34.643 +0000 INFO  loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
01-12-2016 15:45:34.643 +0000 INFO  loader - Using cipher suite TLSv1+HIGH:@STRENGTH
01-12-2016 15:45:34.643 +0000 INFO  loader - ECDH curve not configured
01-12-2016 15:45:34.803 +0000 INFO  SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-12-2016 15:45:34.803 +0000 INFO  SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-12-2016 15:45:34.803 +0000 INFO  SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-12-2016 15:45:34.804 +0000 WARN  ClusteringMgr - Ignoring clustering configuration, the active license disables this feature.
01-12-2016 15:45:34.805 +0000 INFO  SHPoolingMgr - shpooling disabled
01-12-2016 15:45:34.805 +0000 INFO  DS_DC_Common - Initializing the PubSub system.
01-12-2016 15:45:34.805 +0000 INFO  DS_DC_Common - Initializing core facilities of PubSub system.
01-12-2016 15:45:34.813 +0000 INFO  DC:DeploymentClient - target-broker clause is missing.
01-12-2016 15:45:34.813 +0000 WARN  DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Client not initialized.
01-12-2016 15:45:34.813 +0000 INFO  DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-12-2016 15:45:34.813 +0000 INFO  IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-12-2016 15:45:34.814 +0000 INFO  IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory).  This is normal when splunk is first starting up.
01-12-2016 15:45:34.815 +0000 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 15:45:34.815 +0000 INFO  IndexProcessor - running splunkd specific init
01-12-2016 15:45:34.816 +0000 INFO  loader - Initializing from configuration
01-12-2016 15:45:34.817 +0000 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 15:45:34.819 +0000 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 15:45:34.819 +0000 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 15:45:34.966 +0000 INFO  TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 15:45:34.966 +0000 INFO  ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 15:45:34.971 +0000 INFO  TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO  PipelineComponent - Launching the pipelines.
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:45:34.973 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 15:45:35.018 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:45:35.031 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997

@outcoldman
Nothing yet... Although I don't see any errors in the logs...
01-12-2016 15:55:02.774 +0000 INFO  loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - TailWatcher initializing...
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:55:02.841 +0000 INFO  TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:55:02.841 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:55:02.853 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 15:55:32.622 +0000 WARN  AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-12-2016 15:55:32.975 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 15:56:02.927 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 15:56:33.254 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 15:57:03.284 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997
01-12-2016 15:57:33.315 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997

The container can see the current logs in the /var/log/messages...
root@51e26f57eb48:/opt/splunk# tail -f /var/log/messages
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.690Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 301: change for package lodash._isiterateecall"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.693Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 302: change for package lodash.templatesettings"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.696Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 303: change for package lodash.escape"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.697Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 304: change for package lodash._getnative"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.702Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 305: change for package lodash.keys"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.706Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 306: change for package lodash.isarray"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.709Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 307: change for package lodash.isarguments"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.710Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 308: change for package clone-stats"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.729Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 309: change for package clone"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.732Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 310: change for package flagged-respawn"}

Nothing significant found in the logs either... :(
root@51e26f57eb48:/opt/splunk# ls -la var/log/splunk/
total 92
drwx------ 2 root root  4096 Jan 12 16:01 .
drwx--x--x 4 root root    39 Jan 12 15:55 ..
-rw------- 1 root root   140 Jan 12 15:55 audit.log
-rw------- 1 root root     0 Jan 12 15:55 btool.log
-rw------- 1 root root   307 Jan 12 16:00 conf.log
-rw------- 1 root root    64 Jan 12 15:55 first_install.log
-rw------- 1 root root     0 Jan 12 15:55 license_audit.log
-rw------- 1 root root     0 Jan 12 15:55 license_usage.log
-rw------- 1 root root 47883 Jan 12 16:01 metrics.log
-rw------- 1 root root     0 Jan 12 15:55 mongod.log
-rw------- 1 root root     0 Jan 12 15:55 remote_searches.log
-rw------- 1 root root     0 Jan 12 15:55 scheduler.log
-rw------- 1 root root     0 Jan 12 15:55 searchhistory.log
-rw------- 1 root root     0 Jan 12 15:55 splunkd_access.log
-rw------- 1 root root 20262 Jan 12 16:01 splunkd.log
-rw------- 1 root root    61 Jan 12 15:55 splunkd_stderr.log
-rw------- 1 root root     0 Jan 12 15:55 splunkd_stdout.log
-rw------- 1 root root     0 Jan 12 15:55 splunkd_ui_access.log
-rw------- 1 root root  1125 Jan 12 15:55 splunkd-utility.log
root@51e26f57eb48:/opt/splunk# cat var/log/splunk/splunkd_stderr.log
2016-01-12 15:55:02.408 +0000 splunkd started (build 271043)

Maybe turning on DEBUG mode of the forward can show something else?

@marcellodesales sure we can turn on DEBUG for something, but I have no clue where we lose data.
Another suggestion - try to use splunk standalone instance just to try if you will get these messages indexed inside Splunk without actually forwarding them.

@outcoldman
Suddenly, things started working this time, as I see it picked up the files...
01-12-2016 16:16:49.475 +0000 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 16:16:49.494 +0000 INFO  WatchedFile - Will begin reading at offset=19716950 for file='/var/log/messages'.
01-12-2016 16:16:49.497 +0000 INFO  WatchedFile - Will begin reading at offset=1125 for file='/opt/splunk/var/log/splunk/splunkd-utility.log'.
01-12-2016 16:16:49.499 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/btoo
l.log'.
01-12-2016 16:16:49.502 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/sear
chhistory.log'.
01-12-2016 16:16:49.503 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_access.log'.
01-12-2016 16:16:49.505 +0000 INFO  WatchedFile - Will begin reading at offset=48771 for file='/opt/splunk/var/log/splunk/audit.log'.
01-12-2016 16:16:49.507 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/mong
od.log'.
01-12-2016 16:16:49.509 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/lice
nse_usage.log'.
01-12-2016 16:16:49.510 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/lice
nse_audit.log'.
01-12-2016 16:16:49.513 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/remo
te_searches.log'.
01-12-2016 16:16:49.515 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/sche
duler.log'.
01-12-2016 16:16:49.516 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_ui_access.log'.
01-12-2016 16:16:49.517 +0000 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_stdout.log'.
01-12-2016 16:16:49.519 +0000 INFO  WatchedFile - Will begin reading at offset=61 for file='/opt/splunk/var/log/splunk/splunkd_stderr.log'.
01-12-2016 16:16:49.558 +0000 INFO  TcpOutputProc - Connected to idx=<IP_ADDRESS>:9997

The logs event date in Splunk are using a different time...
root@51e26f57eb48:/opt/splunk# date
Tue Jan 12 16:18:49 UTC 2016
root@51e26f57eb48:/opt/splunk# tail -f /var/log/messages
Jan 12 08:21:44 pe2enpmas300 docker/policyfollower[1358]: storing document admn at 640999
Jan 12 08:21:44 pe2enpmas300 docker/policyfollower[1358]: done processing admn at 640999
Jan 12 08:21:45 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:45.186Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:46 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:46.533Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:16:21:46 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:47 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:47.189Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:47 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:47.335Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:16:21:47 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 12 08:21:49 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:49.192Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:51 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:51.195Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:51 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:51.467Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:16:21:51 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:52 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:52.385Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:16:21:52 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 12 08:21:53 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:53.197Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:55 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:55.200Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:56 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:56.525Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:16:21:56 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:57 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:57.203Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:57 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:57.302Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"<IP_ADDRESS> - - [12/Jan/2016:16:21:57 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}

Where there's of offset in the Web UI of Less 8 hours... We are very close now :+1:

How can I correlate the Time column with the time displayed in the Event column???
I noticed that the date in the container is UTC
root@51e26f57eb48:/opt/splunk# date
Tue Jan 12 16:26:24 UTC 2016

@marcellodesales do you mean that it shows you wrong time or not expected TimeZone? I guess the difference of 8 hours is expected if your indexers show you time in your timezone and forwarder collects everything in UTC. If you are in Pacific Time zone - looks like everything is working.

@outcoldman Everything is working!!! The only problem is that we are in the PST timezone, but the Time shown in the Time column is less 8 hrs... Is there a way to set the forwarder to collect everything in PST instead?
I'm tring to set the time inside the container, but no luck so far... What I tried was:

Set the environment var TZ
Mount the /etc/timezone

Mounting the timezone did not work because my host is RHEL7 and I did not find the same files in your container...  (not sure...)

@marcellodesales no need to modify anything on system, in logs everything looks good, you have timestaps in UTC timezone and they show the right time.
Please read http://docs.splunk.com/Documentation/Splunk/6.2.8/Data/Configuretimestamprecognition
Take a look on etc/system/default/props.conf definition for syslog
[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF

As you can see it does not have TZ, so add props.conf inside etc/system/default/props.conf
[syslog]
TZ = UTC

And actually you have better time inside of the syslog message - I would recommend to set timezone parsing of these values instead of timestamps from syslog.

@outcoldman Sounds good and thanks for all your help... The deployment on the second host worked... Here's what I have so far:
splunkforwarderData:
  image: busybox
  volumes:
    - ./monitor/splunk:/opt/splunk/etc/system/local
    - /var/log/messages:/var/log/messages:ro
    - /usr/share/zoneinfo/America/Los_Angeles:/etc/localtime:ro

The server.conf without any SSL setting... Since the container needs to write the values for the server, the volume is mounted with write permission...
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
serverName = pe2enpmas300.company.intuit.net

Thanks a lot for more pointers... I will verify with OPS how the proceed with that...

:+1:


Bot Framework V4 Quicker Connect + Endpoint suggesting

Hi All,
First of all, loving the new UI! looks slick and clean and allows for multiple bot configurations - which I love a lot :)
I am however missing a couple of things that I would maybe like to see added if possible?

Instant connection: In the old Bot Framework Emulator we could just click the blue bar, and http://localhost:3978/api/messages would automatically pop up, allowing us to connect instantly for development.
Suggestion for the correct endpoint? When I start the bot, it doesn't fill in anything for me. Could we maybe add a placeholder field showing http://localhost:3978/api/messages as default endpoint? :)

Thanks for this feedback. This is an interesting counterpoint to the behavior we saw from new users in usability studies: with the endpoint field prefilled, users didn't pay any attention to the endpoint URL and therefore didn't realized they needed to change it when using a sample bot that used a different endpoint. Because of that we chose to remove the default endpoint. But we'll keep this in mind while we're refining the UX.

@thebillkidy - we've made considerable investments on both these pieces of feedback in the latest emulator release -
1 - you can now very easily create a new bot configuration and connect to your bot. You can also easily open saved bot configurations via the MRU list in the Welcome page as well as via File -> open menu.
2. We've also added hint text for /api/messages in the new bot configuration dialog.
Marking this as closed. Please feel free to reactivate or file a new ticket with other suggestions you may have.
Thank you!


Missing contact information on the repo.

SubstrateNetApi is written in NETStandard2.0 to provide maximum compatibility for Unity3D. Feedback, constructive critisme and disscussions are welcome and will help us to improve the API!

Where do we have discussions or criticism?
I am interested in using this repo more seriously but not sure where to discuss technical details.

Got creative with Google:
https://dotmog.com/

You can discuss it here or contact us directly<EMAIL_ADDRESS>


First commit in an empty repo.

These are the step to reproduce this issue.

create an empty repo using Repo.create
add some files to this folder
make a commit

then the error occured:
Error Domain=org.libgit2.libgit2 Code=-3 "reference 'refs/heads/master' not found" UserInfo={NSLocalizedFailureReason=git_reference_name_to_id failed., NSLocalizedDescription=reference 'refs/heads/master' not found}

Since there is no commit yet, the reference(refs/heads/master) is not generated. So the git_reference_name_to_id method just failed.
I tried to create a ref in public func commit(message: String, signature: Signature) using git_reference_create, but then I got:
Error Domain=org.libgit2.libgit2 Code=-3 "odb: cannot read object: null OID cannot exist" UserInfo={NSLocalizedFailureReason=git_object_lookup failed., NSLocalizedDescription=odb: cannot read object: null OID cannot exist}

I don't have any clue with this error. I'm not very familiar with either swift or c. So could someone help resolving this issue?

I too am facing this issue. I would be happy to contribute back examples once I have it working!
@INCHMAN1900 did you fix this for yourself?

I will need to feature in a month or two. If this is still unresolved by then, I would be happy to add this functionality.

The parentID is empty in the commit. I fixed this in Repository.swift by checking on parentID using git_oid_iszero(&parentID) as follows:
        /// Perform a commit of the staged files with the specified message and signature,
	/// assuming we are not doing a merge and using the current tip as the parent.
	public func commit(message: String, signature: Signature) -> Result<Commit, NSError> {
		return unsafeIndex().flatMap { index in
			defer { git_index_free(index) }
			var treeOID = git_oid()
			let treeResult = git_index_write_tree(&treeOID, index)
			guard treeResult == GIT_OK.rawValue else {
				let err = NSError(gitError: treeResult, pointOfFailure: "git_index_write_tree")
				return .failure(err)
			}
			var parentID = git_oid()
			if git_oid_iszero(&parentID) == 1 {
				// The parent is nil, so just commit the tree without parentCommits
				return commit(tree: OID(treeOID), parents: [], message: message, signature: signature)
			} else {
				let nameToIDResult = git_reference_name_to_id(&parentID, self.pointer, "HEAD")
				guard nameToIDResult == GIT_OK.rawValue else {
					return .failure(NSError(gitError: nameToIDResult, pointOfFailure: "git_reference_name_to_id"))
				}
				return commit(OID(parentID)).flatMap { parentCommit in
					commit(tree: OID(treeOID), parents: [parentCommit], message: message, signature: signature)
				}
			}
		}
	}

I will probably prep a PR but thought I'd drop this here now in case it's helpful to someone. Maybe there are cases where this is a bad idea, I am not sure yet.


Kubernetes deployment

Created the necessary Kubernetes metadata files for deployment (deployment.yaml, service.yaml,postgres.yaml) for #65
Currently, the deployment is active and running at dev namespace.
Once the pipeline is setup on IBM Cloud, we'll have automatic deployment to Prod and Dev Namespace as discussed on Slack with Professor.

Looks good.