Removing PopupMenu and CanvasLayer connected by set_custom_viewport crashes Godot

Godot version
4.0.alpha.custom_build. ae2d53cdc
System information
Ubuntu 21.10 - Nvidia GTX 970, Gnome shell 40.4 X11
Issue description
When executing
func _process(delta):
	var temp_variable1763 = CanvasLayer.new()
	add_child(temp_variable1763)

	var temp_argument1763_f10_0 = PopupMenu.new()
	temp_variable1763.set_custom_viewport(temp_argument1763_f10_0)
	temp_argument1763_f10_0.queue_free()

	for i in get_children():
		i.queue_free()

Godot crashes with this backtrace
==24301==ERROR: AddressSanitizer: heap-use-after-free on address 0x61d000229290 at pc 0x000010936f3a bp 0x7ffcb8ac5b00 sp 0x7ffcb8ac5af0
READ of size 8 at 0x61d000229290 thread T0
    #0 0x10936f39 in CanvasLayer::_notification(int) scene/main/canvas_layer.cpp:178
    #1 0x1094a97d in CanvasLayer::_notificationv(int, bool) scene/main/canvas_layer.h:38
    #2 0x1a77bcf7 in Object::notification(int, bool) core/object/object.cpp:848
    #3 0x109b2463 in Node::_propagate_exit_tree() scene/main/node.cpp:292
    #4 0x109b1e54 in Node::_propagate_exit_tree() scene/main/node.cpp:283
    #5 0x109b1e54 in Node::_propagate_exit_tree() scene/main/node.cpp:283
    #6 0x10a00b68 in Node::_set_tree(SceneTree*) scene/main/node.cpp:2468
    #7 0x10ae58c2 in SceneTree::finalize() scene/main/scene_tree.cpp:564
    #8 0x23bfc18 in OS_LinuxBSD::run() platform/linuxbsd/os_linuxbsd.cpp:362
    #9 0x23acd6f in main platform/linuxbsd/godot_linuxbsd.cpp:68
    #10 0x7f4212981fcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #11 0x7f421298207c in __libc_start_main_impl ../csu/libc-start.c:409
    #12 0x23ac82d in _start (/usr/bin/godot4s+0x23ac82d)

0x61d000229290 is located 16 bytes inside of 2344-byte region [0x61d000229280,0x61d000229ba8)
freed by thread T0 here:
    #0 0x7f421395640f in __interceptor_free ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:122
    #1 0x193353b6 in Memory::free_static(void*, bool) core/os/memory.cpp:168
    #2 0x256df5c in void memdelete<Object>(Object*) core/os/memory.h:111
    #3 0x10af37a2 in SceneTree::_flush_delete_queue() scene/main/scene_tree.cpp:1015
    #4 0x10ae2431 in SceneTree::process(double) scene/main/scene_tree.cpp:458
    #5 0x2556ec0 in Main::iteration() main/main.cpp:2722
    #6 0x23bf700 in OS_LinuxBSD::run() platform/linuxbsd/os_linuxbsd.cpp:357
    #7 0x23acd6f in main platform/linuxbsd/godot_linuxbsd.cpp:68
    #8 0x7f4212981fcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

previously allocated by thread T0 here:
    #0 0x7f4213956808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x1933435b in Memory::alloc_static(unsigned long, bool) core/os/memory.cpp:75
    #2 0x1933426c in operator new(unsigned long, char const*) core/os/memory.cpp:40
    #3 0x1080ed40 in Object* ClassDB::creator<PopupMenu>() core/object/class_db.h:129
    #4 0x1a6f376f in ClassDB::instantiate(StringName const&) core/object/class_db.cpp:523
    #5 0x863d980 in GDScriptNativeClass::instantiate() modules/gdscript/gdscript.cpp:92
    #6 0x863d361 in GDScriptNativeClass::_new() modules/gdscript/gdscript.cpp:80
    #7 0x8723a79 in void call_with_variant_args_ret_helper<__UnexistingClass, Variant>(__UnexistingClass*, Variant (__UnexistingClass::*)(), Variant const**, Variant&, Callable::CallError&, IndexSequence<>) core/variant/binder_common.h:654
    #8 0x87234d4 in void call_with_variant_args_ret_dv<__UnexistingClass, Variant>(__UnexistingClass*, Variant (__UnexistingClass::*)(), Variant const**, int, Variant&, Callable::CallError&, Vector<Variant> const&) core/variant/binder_common.h:467
    #9 0x8722bae in MethodBindTR<Variant>::call(Object*, Variant const**, int, Callable::CallError&) core/object/method_bind.h:495
    #10 0x8bd765b in GDScriptFunction::call(GDScriptInstance*, Variant const**, int, Callable::CallError&, GDScriptFunction::CallState*) modules/gdscript/gdscript_vm.cpp:1633
    #11 0x8674ef5 in GDScriptInstance::callp(StringName const&, Variant const**, int, Callable::CallError&) modules/gdscript/gdscript.cpp:1539
    #12 0x10a39925 in bool Node::_gdvirtual__process_call<false>(double) scene/main/node.h:229
    #13 0x109a4f44 in Node::_notification(int) scene/main/node.cpp:56
    #14 0x3057457 in Node::_notificationv(int, bool) scene/main/node.h:45
    #15 0x1a77bcf7 in Object::notification(int, bool) core/object/object.cpp:848
    #16 0x10aef3b5 in SceneTree::_notify_group_pause(StringName const&, int) scene/main/scene_tree.cpp:851
    #17 0x10ae1ea5 in SceneTree::process(double) scene/main/scene_tree.cpp:450
    #18 0x2556ec0 in Main::iteration() main/main.cpp:2722
    #19 0x23bf700 in OS_LinuxBSD::run() platform/linuxbsd/os_linuxbsd.cpp:357
    #20 0x23acd6f in main platform/linuxbsd/godot_linuxbsd.cpp:68
    #21 0x7f4212981fcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

This example was found by Godot fuzzer - Qarminer, so it is quite unlikelly that this code could be used in real project, but still this should be handled gracefully.
Steps to reproduce
Above
Minimal reproduction project
No response

This is very similar issue to https://github.com/godotengine/godot/issues/53445, so maybe fix will be also similar

Seems fixed in 4.1.
Spams this error, but no (immediate) crash:
ERROR: Cannot disconnect 'child_order_changed' from callable 'Viewport::canvas_parent_mark_dirty': the callable object is null.
   at: _disconnect (./core/object/object.cpp:1343)
ERROR: Condition "!viewport" is true.
   at: viewport_remove_canvas (./servers/rendering/renderer_viewport.cpp:1074)


Getting the error value without using forceDecode (exceptions)

Hi,
Thanks for open sourcing this little library, I like the minimal api, it provides just the right basics.
Right now decode returns null | T , to get the error we need to use forceDecode and then catch the exception.
I'm wondering if we could have another function validate which doesn't throw an exception but instead returns a union of error and data.
validate(data: unknown) => {type: 'Error', message: String} | {type: 'Ok', data: T}

The above api is easy to pattern match using a switch case.

I like this! I have a busy week at work though. I will get to this eventually in the second half of the week.
However, if you feel like you're welcome to make a PR. It's not hard!

Closed with @aszenz nailing #10.


Try to use the base project gpt-pdf-summarizer

Why
To see whether to upload pdfs and summarize them on https://github.com/zerebom/gpt-pdf-summarizer project
Definition of Done

Upload some pdfs
Download summarized pdfs
Check and analyze the detailed code of the project above
Results will be reported under this issue.

How

Try it out on the project page

Report
At first, I met an error of the openai.api_key not keeping the API key value.  But the code finally worked by hardcoding it.
The repo itself seems to be made under the instruction of chatGPT, meaning that it has redundant or unnecessary parts.
Also, as their prompt is pretty simple (i.e. 下記文章を日本語で{max_length}字で要約して:\n\n{text}\n), we need to think up the better prompt to generate the better summarization.
For the token length problem, they just split the paper text into some chunks and generate summarization by applying the prompt above repeatedly.  If we have access to the GPT4, this would not be an issue.

Thank you for summarizing your experiments! It's really helpful! About prompt part, let's do "prompt engineering" in the future step.
I think now this ticket is done. So let me close this issue.


The tooltips starting with help are not resolved properly

          The tooltips starting with `help` are not resolved properly in our instance:

Originally posted by @rekt-hard in https://github.com/tugraz-rdm/damap-frontend/pull/31#pullrequestreview-1996337246

@rekt-hard please add also this part here since its related. Thanks
Localization in DE for DMP TUGraz
Problem: German localization issues observed specific to DMP TUGraz.

Also info.ethical.humanParticipants.researchEthics.text


Cancelling a workflow instance

When a workflow with a finished timer is canceled, we run into an invalid state transition for the timer command.
Should probably remove the callback once the command is done, or check if the state of the command is cancelable before attempting to do so.

This is a similar problem.
I have a MainWorkflow with 2 SubWorkflow like this:
If I cancel the MainWorkflow by CancelWorkflowInstance before SubWorkflow2 has finished(in this time SubWorkflow1 has finished), there will be a panic raised.
"invalid state transition for command ScheduleSubWorkflow: Done -> Canceled"
package entity

import (
	"context"
	"github.com/cschleiden/go-workflows/workflow"
	"github.com/google/uuid"
	"sync"
	"time"
)

func MainWorkflow(ctx workflow.Context, input string) (string, error) {
	println("MainWorkflow start")

	fs := make([]workflow.Future[string], 0, 2)
	fs = append(fs, workflow.CreateSubWorkflowInstance[string](ctx, workflow.SubWorkflowOptions{InstanceID: uuid.NewString()}, SubWorkflow1, "SubWorkflow1"))
	fs = append(fs, workflow.CreateSubWorkflowInstance[string](ctx, workflow.SubWorkflowOptions{InstanceID: uuid.NewString()}, SubWorkflow2, "SubWorkflow2"))

	var wg sync.WaitGroup

	for i := range fs {
		wg.Add(1)
		go func(i int) {
			defer wg.Done()
			f := fs[i]
			workflow.Select(ctx, workflow.Await(f, func(c workflow.Context, f workflow.Future[string]) {
				res, err := f.Get(ctx)
				println(res)
				println(err)
			}))
		}(i)
	}

	wg.Wait()

	return "succeed MainWorkflow", nil
}

func SubWorkflow1(ctx workflow.Context, input string) (string, error) {
	return workflow.ExecuteActivity[string](ctx, workflow.DefaultActivityOptions, Activity1).Get(ctx) // dipatch cluster1
}

func SubWorkflow2(ctx workflow.Context, input string) (string, error) {
	return workflow.ExecuteActivity[string](ctx, workflow.DefaultActivityOptions, Activity2).Get(ctx)
}

func Activity1(ctx context.Context) (string, error) {
	return "Activity1 ok", nil
}

func Activity2(ctx context.Context) (string, error) {
	time.Sleep(30 * time.Second)
	return "Activity2 ok", nil
}


Unit tests for API

[ ] empty
[ ] singleton
[ ] fromList
[ ] test
[ ] pop
[ ] null
[ ] rank
[ ] select
[ ] bits
[ ] isSubsetOf
[ ] set
[ ] clear
[ ] difference
[ ] union
[ ] intersection
[ ] toList
[ ] toBits
[ ] toVector

@thsutton commenting here - is there interest in continuing this project? stumbled across roaring bitmaps and i wanted to learn (about the bitmap and haskell) so was looking to contribute over here, if you're willing!


Soft Wrap Hanging Indent not working

I'm trying to enable the softWrapHangingIndent setting. It was added in #6003, but it doesn't seem to be working for me. Here's what my wrapping settings look like:

With these settings enabled, here's some HTML I pulled from Hipster Ipsum:

This is what I expect it to look like:

Am I missing something? Thanks in advance!

@LandonSchropp That works for me:

Can you reliably reproduce this? If so, can you post the exact instructions which reproduce this for you? Also, which version of Atom and OS are you using, and can you reproduce this in safe mode?

I was able to reproduce this on 0.182 in safe mode on OS X 10.10.2, but I installed the update and the problem seems to be fixed. Thanks for the help!


It appears that markdownItOptions does not support the highlight function

If you could take a look and fix some

@jekip I don't believe that is part of the core API is it? API Docs. I think what you're looking for is the result of using a plugin to Markdown-IT. Maybe Prism or Highlighter.js?

@jekip I don't believe that is part of the core API is it? API Docs. I think what you're looking for is the result of using a plugin to Markdown-IT. Maybe Prism or Highlighter.js?

you're right Finally, the solution was combined with Highlighter


chore: Added "Start Contribution" with "Star on GitHub" Feature 🌟🚀

Related Issue
Issue : #1810
Description
This PR proposes an improved logo type for the ProjectHunt Project to enhance user engagement and encourage GitHub contributions. The updated logo prominently showcases "Start Contribution" ✨ at the top, followed by "Star on GitHub" 🤝 to prompt users to interact with the repository. Additionally, a dynamic element indicates the total number of stars the project has received, giving users a quick overview of its popularity 🌟.
Key Changes:

Responsive Design: The logo is now responsive, ensuring a seamless experience across various screen sizes.

Code Changes:
<div className="github-button p-2 bg-white rounded-lg max-w-[170px] mx-auto">
  <a
    className="github"
    href="https://github.com/priyankarpal/ProjectsHut"
    target="_blank"
    rel="noreferrer"
  >
    <img
      src="https://github.com/priyankarpal/ProjectsHut/assets/88102392/0fa9e8f0-349d-4cc8-8e01-1b6e2c27dbbb"
      alt="Github-Share-Button"
    />
  </a>
</div>

Screenshots

https://github.com/priyankarpal/ProjectsHut/assets/85815172/82e2caa9-171a-4432-8f36-ccb3ec767ce9
@priyankarpal Please check the changes

I have fix alignment

@priyankarpal Check

@priyankarpal Fixed you reviews

@Nishitbaria in the future, kindly refrain from using these types of commit messages, as they do not adhere to the valid format. If a user reviews the repository, they might encounter difficulties in understanding the changes made by the contributor.

@priyankarpal Man I deserve level 2 atleast

@priyankarpal Man I deserve level 2 atleast lol 😂

I believe that it was a simple change 🙂

@priyankarpal Ohh I see Thanks for Merging 🙃


Worker Timeout

Hi there,
I may well be doing something wrong, but having issues with timeouts.. System works fine for a couple of minutes, then randomly stops - this halts fetching the file from the server and makes web pages hang.
I am running an unusual setup, using two VMs. One is my main ingress via a Caddy reverse proxy, which then sends the request internally to the ShyNet server which is on a different machine.
I don't think this should affect operation - any thoughts?

Launching Shynet web server...
[2020-04-28 21:45:16 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-04-28 21:45:16 +0000] [1] [INFO] Listening at: http://<IP_ADDRESS>:8080 (1)
[2020-04-28 21:45:16 +0000] [1] [INFO] Using worker: sync
[2020-04-28 21:45:16 +0000] [8] [INFO] Booting worker with pid: 8
[2020-04-28 21:45:16 +0000] [9] [INFO] Booting worker with pid: 9
[2020-04-28 21:45:16 +0000] [10] [INFO] Booting worker with pid: 10
[2020-04-28 21:47:16 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
[2020-04-28 22:47:16 +0100] [8] [INFO] Worker exiting (pid: 8)
[2020-04-28 21:47:16 +0000] [11] [INFO] Booting worker with pid: 11
[2020-04-28 21:47:18 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:10)
[2020-04-28 22:47:18 +0100] [10] [INFO] Worker exiting (pid: 10)
[2020-04-28 21:47:19 +0000] [12] [INFO] Booting worker with pid: 12

Thanks in advance - this project looks really promising !

Hmm, this definitely isn't expected behavior, but also not something I've been able to replicate. Would you mind posting a redacted version of your environment file? I have a feeling this might be related to a misconfigured cache.

This Gunicorn issue may be related, though it's still unclear why this would cause requests to hang.

.env contents:

Database
DB_NAME=xyz
DB_USER=xyz
DB_PASSWORD=xyz
DB_HOST=xyz
DB_PORT=xyz
General Django settings
DJANGO_SECRET_KEY=xyz
Don't leak error details to visitors, very important
DEBUG=False
Unless you are using an external Celery task queue, make sure this
is set to True.
CELERY_TASK_ALWAYS_EAGER=True
For better security, set this to your deployment's domain. Comma separated.
ALLOWED_HOSTS=*
Set to True (capitalized) if you want people to be able to sign up for your Shynet instance (not recommended)
SIGNUPS_ENABLED=False
Change as required
TIME_ZONE=Europe/Dublin
Set to "False" if you will not be serving content over HTTPS
SCRIPT_USE_HTTPS=True

All seems okay with the connection between Caddy & ShyNet machines - other services are working fine...
Have your tests all worked fine with 3 workers?

This Gunicorn issue may be related, though it's still unclear why this would cause requests to hang.

There was a response in there about a security group - I added another rule, and the logs have calmed down a bit (there was one that broke through in the mean time) - so it might have been me missing a step on my end.
I’ll get back to you tomorrow with an update. Thanks for the help thus far!

Let me know what you find. The worker threads are intended to be somewhat disposable, so they'll often die for no real reason. Why Gunicorn prints the log as "CRITICAL" is beyond me.
Either way, let me know what you find w.r.t. the security rules and I'll improve the documentation so that others in the future don't have the same issue.

No change here I'm afraid, workers still hanging and causing the site to become unresponsive.
As I can access the ShyNet UI, I assume all the security rules are correctly set on the instances - otherwise I wouldn't get a connection to it.
Also, I'm not actually getting any results on the ShyNet UI for any hits on the site. When the site isn't hanging, it's showing a 200 connection for the GET collecting the js, then a 200 for OPTIONS, but no responses for the POST or any subsequent OPTIONS.
Any thoughts?

It appears like I'm running into the same issue. UI is responsive and works, but the tracking just hangs and there's critical worker timeouts in the logs. Once I've started a tracking request, the UI starts lagging as well.
Fwiw, I'm running Shynet in a docker container (:latest)and Postgres (:12)in an other, both started from docker-compose. There's also an Adminer instance running, through which I can reach and operate the database just fine. No Redis in the mix just yet, but from JXGA's comments, I don't think that'll solve things.
Anything useful I can do to help troubleshoot this? If I enable debug logs, where do those end up?

Alright, the information about the hanging POSTs is very helpful: it made me realize that I've seen this issue before! (Unfortunately, I don't remember exactly how I solved it, but I believe it was a configuration change.)
I am able to replicate this issue by turning CELERY_TASK_ALWAYS_EAGER to False (or by omitting it entirely), but can't seem to figure out a way to replicate this by other means.
@doenietzomoeilijk—would you be able to post a redacted version of your environment as well?

@milesmcc Sure thing!
Here's docker-compose.yml:
version: '2'
services:
  app:
    image: milesmcc/shynet:latest
    container_name: shynet
    env_file:
      - .env
    networks:
      - shynet_net
    ports:
      - 18085:8080
  db:
    image: postgres:12
    env_file:
      - .env
    networks:
      - shynet_net
    environment:
      - POSTGRES_USER=${DB_USER}
      - POSTGRES_PASSWORD=${DB_PASSWORD}
      - POSTGRES_DB=${DB_NAME}
      - PGDATA=/var/lib/postgresql/data/pgdata
    volumes:
      - shynet_postgres_data:/var/lib/postgresql/data
  adminer:
    image: adminer
    restart: unless-stopped
    networks:
      - shynet_net
    ports:
      - 18086:8080
volumes:
  shynet_postgres_data: ~
networks:
  shynet_net:
    driver: bridge

And this is the .env file:
DJANGO_SECRET_KEY=REDACTED
ALLOWED_HOSTS=*
TIME_ZONE=Europe/Amsterdam
DEBUG=True
SIGNUPS_ENABLED=False
DB_USER=shynet
DB_PASSWORD=REDACTED
DB_NAME=shynet
DB_HOST=db
MAIL_HOST=REDACTED
SERVER_EMAIL=noreply@REDACTED

That's it, basically. No mention of Celery, although I could give that a go.
Anything else that could be interesting?

Ah! @doenietzomoeilijk — add CELERY_TASK_ALWAYS_EAGER=False to your env, and also try pulling the new version (v0.3.0). In your case, this should probably fix the issue (even though the original issue remains unresolved).

That doesn't seem to solve it. I pulled the image (which did get me a newer image) and currently have these in my .env:
CELERY_TASK_ALWAYS_EAGER=False
CELERY_BROKER_URL="redis://redis/1"

This results in the script not timing out, but now I see this in my logs:
shynet     | ERROR Internal Server Error: /ingress/008eaa7f-d2dc-49da-87d2-50b6086e3d61/script.js
shynet     | Traceback (most recent call last):
shynet     |   File "/usr/local/lib/python3.8/site-packages/kombu/transport/__init__.py", line 59, in resolve_transport
shynet     |     transport = TRANSPORT_ALIASES[transport]
shynet     | KeyError: ''
shynet     | During handling of the above exception, another exception occurred:

...repeatedly. Does this ring a bell?
I'll start an .env file from the actual example, maybe it's missing a key.

@doenietzomoeilijk — yes, this is expected behavior. Celery is a way to offload processing of certain asynchronous tasks to a separate machine (or a separate cluster of instances). CELERY_TASK_ALWAYS_EAGER is a way to tell Celery "hey, there's no separate separate machine, don't put anything into the queue." The worker won't spawn by default; the point is to do that work elsewhere.
When you run with CELERY_TASK_ALWAYS_EAGER not set to True, Celery puts ingress tasks into the queue instead of doing them in real time. In order to process those tasks, you need to have a separate instance of Shynet running via celeryworker.sh. This worker won't be launched automatically.
I'll put a note about this in GUIDE.md. @JXGA — have you triple checked that CELERY_TASK_ALWAYS_EAGER is being parsed as True, and that your cache is setup correctly? I'm having trouble figuring out what else can be the issue.

@milesmcc OK, that makes sense, sort of. Most single-machine instances wouldn't need the Redis layer, and thereby, Celery, correct? Or rather, the other way around, they don't need the Celery task queue, so they don't need the broker to run it on? I agree that that could do with a bit of documentation, yes.
However, that wouldn't explain the hangs I encountered with no Celery configured, or was that the thing that was changed in the last pull I did? Anyway, I've removed the Celery settings from my .env file and shut down Redis now, as it runs fine without it, and Redis doesn't bring an appreciable speed difference.
So for me, the problem is solved, let's hope it turns out to be something similar for @JXGA !

I have pulled latest and set CELERY_TASK_ALWAYS_EAGER to True, seems to have done the trick! I have always had it set to True from the first instance...
I had a bit of trouble changing the hostname last night (noticed a few changes in the docs) - using the python manage.py hostname "xyz.com" didn't do anything, so opened a shell into the container and ran it within the container - did the trick.
Delighted to have hits showing - thanks for the help both!

Glad it's working for you @JXGA!


Update Cppcheck 2.13.0

Cppcheck 2.13.0 is available.

From the release notes:

Building with Qt5 has been deprecated (it went EOL in May 2023) and will be removed in a future version - please use Qt6 instead.


Add bubbleSort Java

Fixes #[Add issue number here."First steps for issues #IssueNumber]
Language: [Add here in what language you have implemented this data structure or algorithm.]

Bubble sort in java already there.
You can try working on a file that is not present.


Query: Interface based member for owned/weak entityTypes

Same as https://github.com/aspnet/EntityFrameworkCore/issues/17276
We haven't fixed it for Owned/Weak navigations.

This actually works without any changes in 3.0 already because Navigation Expansion removes it.


Error composer require

Hey, I am trying to install this package but getting this error:
Problem 1
- Root composer.json requires still-code/laravel-umami ^4.0 -> satisfiable by still-code/laravel-umami[4.0.0].
- still-code/laravel-umami 4.0.0 requires illuminate/support ^9.0 -> found illuminate/support[v9.0.0, ..., v9.52.5] but these were not loaded, likely because it conflicts with another require.
You can also try re-running composer require with an explicit version constraint, e.g. "composer require still-code/laravel-umami:*" to figure out if any version is installable, or "c

Are you in laravel 10?
It's not supported yet. I'll release an update soon.

Yes! Okay. It'll wait.

released a new version


Project / Fork Status

Since you want to publish this on npm and since the original maintainers seem to have abandoned the project, will this become a real fork, i.e. renamed binary and stuff?

This was the goal, but there are some other "competing" forks.
You can see the discussion here: https://github.com/rlidwka/sinopia/issues/376
I'm open to advice on how to proceed

I'd say just go for it. The other forks aren't too active and the proposals made by the guy associated with IBM, if I remember correctly, have not yet made their way into the original code base.

So this is the real Sinopia successor? Ho can I install it? From github link?

@sullyme yes, you can isntall it from github right now, im working on getting something on npm. node 6 seems to have caused some problems

@trentearl  I am going to chose this project if it works as expected. I think the community needs to lean somewhere.

@trentearl I wonder what is the path this project gonna have, because, now I'm trying to figure out which fork I should use at work. As far I've seen (aside of bugfixes) in both fork, no big changes since last commit on sinopia original. I'm asking because you mention "how to proceed" and that scares me, how can I suggest something that probably tomorrow might ends up as the original? . I agreed with @silkentrance  go for it. Waiting for the npm package. 👍

@juanpicado  Originally tests were failing on older versions of node and node 6. I now have the verdaccio working with node 6 but the older versions of node still fail. I would like to get these older versions working before it is published to npm.
I asked how to proceed because when I originally forked the project there were a few people who expressed they would like to be co-maintainers and they haven't spoken up since then.
I'm hoping to have time to get all tests passing this weekend.

Go verdaccio go!

0.10 pass on travis but not locally on OSX. I think this is just because the tests are running by starting the server in a new process. I am trying to get the server in test to start in the same process.

Why worry about 0.10 and 0.12? 0.10 will no longer be supported at all on 2016-10-01, and 0.12 will be the same on 2016-12-31. It doesn't make any sense to retain support for them going forward. If someone needs a repo for them then they can just use Sinopia, as it will be just as supported.
Deprecate support for them in Veradaccio and move on.

@jsumners  Thanks I didnt realize it was imminent. I should have a version published in the next few days

I guess the same for iojs, still present in the .travis conf file.

I have prepared the release for 1.0.0.
Unfortunately user lonelyclick has published verdaccio without informing anyone. This prevents me from publishing
I have emailed the user asking what there intentions are and if I can be collaborator.

@lonelyclick that's seriously not cool.

@lonelyclick what gives, brah.

I am so sorry, but when unpublish, I found this http://blog.npmjs.org/post/141905368000/changes-to-npms-unpublish-policy， what should i do ?

Transfer ownership by adding @trentearl and removing yourself I suppose -- https://docs.npmjs.com/cli/owner

→ npm owner ls
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info attempt registry request try #1 at 12:49:20 PM
npm http request GET https://registry.npmjs.org/verdaccio
npm http 200 https://registry.npmjs.org/verdaccio
lonelyclick<EMAIL_ADDRESS>trentearl<EMAIL_ADDRESS>npm info ok

done

Because the currently published version is 1.4.0, I'm assuming the maintainers will have to bump the version to a higher version (2.0.0?), and then publish that.
On a side note, I wonder if it will be okay with @lonelyclick, unless he's a trusted maintainer, to maintain ownership alongside @trentearl on the npm package, as it will give lonelyclick publishing/deprecation/unpublishing/etc capabilities, potentially (intentionally or unintentionally) wreaking havoc on the published package.

@lonelyclick  thank you so much!
Can you remove yourself? If you want to be collaborator, I would be happy to allow that after your contributions bring some trust to the project.
However this is not the correct way for you to have permission to the npm package.
@SEAPUNK I agree I have deprecate the 1.4.0 release and published the 2.0.0 release

→ npm owner ls
npm info it worked if it ends with ok
npm info using<EMAIL_ADDRESS>npm info using<EMAIL_ADDRESS>npm info attempt registry request try #1 at 2:25:07 PM
npm http request GET https://registry.npmjs.org/verdaccio
npm http 200 https://registry.npmjs.org/verdaccio
trentearl<EMAIL_ADDRESS>npm info ok

@trentearl done

@lonelyclick Thank you!

@lonelyclick  thanks!

Closed? Where is the future roadmap supposed to be discussed?

@DanielSundberg sorry for a super late reply. But in case of someone is interested on Roadmaps. We have one for the next release.
https://github.com/verdaccio/verdaccio/pull/376


Is it a BUG? callback of tapAsync not work in AsyncSeriesLoopHook

const { AsyncSeriesLoopHook } = require('tapable')

const testhook = new AsyncSeriesLoopHook(['compilation', 'name'])

let count1 = 1
let count2 = 2
let count3 = 2

testhook.tap('plugin1', (compilation, name) => {
  console.log('plugin1', name)
  compilation.sum = compilation.sum + 1
  if(count1 < 1) return;
  count1--;
  return count1;
})

testhook.tapPromise('plugin2', (compilation, name) => {
  return new Promise((resolve, reject) => {
    console.log('plugin2', name)
    setTimeout(() => {
      if(count2<1){
        resolve()
      } else {
        resolve(count2)
      }
      count2--;
    }, 1000)
    compilation.sum = compilation.sum + 1
  })
})

testhook.tapAsync('plugin3', (compilation, name,cb) => {
  console.log('plugin3', name)
  compilation.sum = compilation.sum + 4
  setTimeout(() => {
    if(count3 < 1) {
      cb()
    } else {
      cb(count3);
    }
  }, 2000)
  return count3;
})

const compilation = { sum: 0 }
testhook.callAsync(compilation, 'mike', function(...args){
  console.log('finish', compilation)
  console.log('an error', args)
})

Expect:
when execute cb(count3) and the count3 is not undefined，the hook should restart from the first plugin

You might want to change cb(count3) to cb(null, count3) because the first argument is an Error argument, which cancels the execution and reports an error

oh yes，thank you for your reply


Fix type inference

Fix type inference from issue #6566 and fix queryOptions false positive test because of bad import of QueryClient.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (22caf2a) 0.00% compared to head (6addbb8) 70.81%.
Report is 3 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##           main    #6706       +/-   ##
=========================================
+ Coverage      0   70.81%   +70.81%
=========================================
  Files         0       18       +18
  Lines         0      442      +442
  Branches      0      113      +113
=========================================
+ Hits          0      313      +313
- Misses        0       99       +99
- Partials      0       30       +30

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Doesn't work with the new game version (2612)

Please note: ScriptHookVDotNet is NOT ScriptHookV!
If you get a FATAL error (e.g. FATAL: Unknown game version after a game update), than this project is not responsible and you instead need to wait for ScriptHookV to update.
I acknowledged the above but got the example error when adding SHVDN to my GTAV folder, I saw that the latest release and the source code aren't up to date with the newest game build 2612 (I saw that the source code wasn't up to date after I tried to build it), I tried to build the solution as well but got some errors trying to do that.

Version
Which version of SHVDN was the problem encountered with? 3.4.0
Description
If you add SHVDN to your game with the newest version of SHV, then it will give you the error FATAL: Unknown game version, trying to build the source code also doesn't work as I'm getting the error shown in the screenshot

font{
    line-height: 1.6;
}
ul,ol{
    padding-left: 20px;
    list-style-position: inside;
}

Be aware of that the latest build is 2628, and SHV has not been updated for 2628 yet.Message ID<EMAIL_ADDRESS>

Ohh, okay, didn't see the 2612 or 2628 in the GameVersion.cs in scripting_v3/GTA, so I assumed it wasn't updated yet, I'll wait for SHV to update then

I’m not convinced if this issue is really related to SHVDN. Are you sure you have installed Visual C++ Redistributable for Visual Studio 2019 x64?
By the way, you can detect newer versions with Game.Version in scripts even if the GameVersion enum does not have corresponding values (SHVDN uses the value returned from SHV).

Ah okay, but I do have all requirements installed, also unpacked SHV SDK into sdk/

Ah, do you mean this typical error? In that case, this issue is not related to SHVDN because SHV currently supports 1.0.2618.1, but doesn't support 1.0.2618.2.

Yeah I got that, but not the build error I got, do you get that or does it build fine for you?

It builds fine for me. The way I see it from your screenshot, I guess you need to install Windows SDK. Make sure 10.0.18362.0 of Windows SDK is available.
Note that building SHVDN from the source doesn’t make difference if you can launch the exe with SHV.
https://github.com/crosire/scripthookvdotnet/blob/ccdf0c1b13688b52c36cf3322da1d15eaea7bc4c/ScriptHookVDotNet.vcxproj#L20

That could be the problem, well thanks for the support!


Corrected docker run command in Readme file.

Instead of requesting from basic.json, it should be from request.json.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

thanks for the correction!


NotebookApp version needing upgrade  when connecting to local runtime

Please excuse me and direct me to another source for information if there is one (couldn't find anything on this)
I am trying to connect to jupyter notebook on google colab's local runtime and when I try to I get this message:
[E 13:51:13.307 NotebookApp] Rejecting connection: Requested version (0.0.1a2) >
Current version (0.0.1a1). Please upgrade this package.
I was running local runtime jobs perfectly fine this morning but a couple hours later it suddenly stopped working. I don't believe I even touched the machine in between this timeframe.
This was the entire message:
(base) C:\Users\ballcap>jupyter notebook \ --NotebookApp.allow_origin='https://col
ab.research.google.com' \ --port=8889
jupyter_http_over_ws extension initialized. Listening on /http_over_websocket
[I 13:50:46.680 NotebookApp] JupyterLab alpha preview extension loaded from C:\U
sers\2hinc\AppData\Local\Continuum\anaconda3\lib\site-packages\jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 13:50:46.727 NotebookApp] Running the core application with no additional ext
ensions or settings
[I 13:50:46.758 NotebookApp] Serving notebooks from local directory: C:\
[I 13:50:46.773 NotebookApp] The Jupyter Notebook is running at:
[I 13:50:46.789 NotebookApp] http://localhost:8889/?token=0af42d8c7656bc5a04fea5
0ea6623ae8bb099209d086e31b
[I 13:50:46.805 NotebookApp] Use Control-C to stop this server and shut down all
 kernels (twice to skip confirmation).
[C 13:50:46.836 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8889/?token=0af42d8c7656bc5a04fea50ea6623ae8bb099209d08
6e31b
[I 13:50:47.007 NotebookApp] Accepting one-time-token-authenticated connection f
rom ::1
[E 13:51:13.307 NotebookApp] Rejecting connection: Requested version (0.0.1a2) >
 Current version (0.0.1a1). Please upgrade this package.

You'll need to update the version of the jupyter_http_over_ws extension you're using. Try running the following and then restart your Jupyter server:
pip install --upgrade jupyter_http_over_ws


Upgrade based on v0.4.2

Sorry for sending PRs to the main fork, I didn't intend to >.<

Never mind @rhaps0dy , by the way, are u using kueue for AI scenarios?

Sure am! It's pretty good, thank you for contributing to it.

Thanks for the feedback, if possible and be willing to, you can fill the https://kueue.sigs.k8s.io/docs/adopters/ for another adopter, we're collecting adopters now.


Excel export specific rationale

Passing specific rationale to the backend when exporting to excel and changing the TRUE, FALSE values based on the specific rationale.

I tried running this using the simple specific occurrence example I drafted a while ago. I got the following error dialog:

The server showed:
ArgumentError (wrong number of arguments (3 for 2)):
  lib/patient_export.rb:16:in `export_excel_file'
  app/controllers/patients_controller.rb:109:in `excel_export'

The patients I created were:
P1

encounter during MP
encounter outside of MP with a diagnosis

P2

encounter during MP with a diagnosis

P3

encounter during MP

In the export, it looks like something that is specifically false isn't differentiated from something that's false. What about adding a differentiator, like an asterisk ("**FALSE")?


Update serde_yaml to v0.8.25

Automated changes by create-pull-request GitHub action

Codecov Report

Merging #33 (2cd31a0) into main (bc926ff) will decrease coverage by 0.09%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
- Coverage   86.06%   85.97%   -0.10%
==========================================
  Files          43       43
  Lines        4414     4414
==========================================
- Hits         3799     3795       -4
- Misses        615      619       +4

Impacted Files
Coverage Δ

src/steps/parse_bin_game_step.rs
92.75% <0.00%> (-1.45%)
:arrow_down:

src/steps/sum_reduce.rs
93.69% <0.00%> (-0.91%)
:arrow_down:

src/macros.rs
95.42% <0.00%> (-0.66%)
:arrow_down:

src/generic_steps/generic_filter.rs
99.59% <0.00%> (-0.21%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bc926ff...2cd31a0. Read the comment docs.


Webhook connection problems shortly after cert-manager installation

I have the described issues installing via script.
If I wait manually wait a minut - there is no problem installing, but via a script I get the error
How to wait for cert-manager - before using it creating issuers etc?
I have tried the following - no luck:
kubectl wait --for=condition=ready --timeout=60s --all-namespaces --all pods
kubectl wait deployment --all --timeout=-1s --for=condition=Available -n cert-manager
kubectl wait --for=condition=ready --timeout=60s -n cert-manager --all pods

Hi @JCzz
I haven't used this myself, but there is https://github.com/alenkacz/cert-manager-verifier let us know if that works for you.
/cc @wallrj @inteon related to problems that could be solved by install/verify commands 😄

Thanks, I will give it a try and let you know

Works


kea: mangle store paths in config.report

Before the change closure size is:
$ nix path-info -rsSh $(nix-build --no-link -A kea) | nl | unnix | tail -n 1
   131  /<<NIX>>/kea-2.4.0 32.0M  405.6M

After the change:
$ nix path-info -rsSh $(nix-build --no-link -A kea) | nl | unnix | tail -n 1
    37  /<<NIX>>/kea-2.4.0 32.0M  246.3M

Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Cool, thanks!


bug: fsspec output filepath including base directory

Description
When passing in a remote path for fsspec-based source connectors, the base directory was always being included in the output path itself. This was updated to exclude the base directory any only include any child directories relative to the base one.

awesome! should require bump to test fixtures, right? since this updates the output path.

changes look good though!

I've tested this manually with the s3 connector using:

A specific file as the input path
An input path with and without the leading slash
Recursive input with subfolders

I just ran with box. The nesting doesn't seem to be working right. I'll have to run against the old version again, but nested-2 should be in nested-1 but now nested-2 is next to nested-1.

Well. Looks like the directory and files in Box got shifted around sometime around 9/10. So it's not nested anyway and there is a partitioned file in the Box folder. So I can't make a conclusion with Box. I'll try to fix the files in Box.com if I can later.
I will try dropbox to see if the nesting works. :)

Dropbox is the same. I can tell from expected-structured-output. No double nesting. Same with GCS. No double nesting.

Works as expected.
Ok. I fixed the files in Box.com and it all works good. You will need to export OVERWRITE_FIXTURES=true for Box.

hmm, what do you mean by that?

Dropbox is the same. I can tell from expected-structured-output. No double nesting. Same with GCS. No double nesting.

were we every doubly nesting these? I don't remember us explicitly doing that? but also if we were, I don't remember us undoing that at any point?

Works as expected.
Ok. I fixed the files in Box.com and it all works good. You will need to export OVERWRITE_FIXTURES=true for Box.

if you make changes to any source test files, please create a quick PR to explicitly fix the expected values. in this case, I think we're still ignoring Box tests so slightly less urgent, but otherwise this could be breaking tests for other active PRs.
I'll go ahead and kick one off now.
also, what changes did you make to the source? we should probably discuss per the "double nested" conversation, since these file storage services are all treated similarly may be worth making sure we're testing consistently.

Works as expected.
Ok. I fixed the files in Box.com and it all works good. You will need to export OVERWRITE_FIXTURES=true for Box.

@potter-potter if you make changes to any source test files, please create a quick PR to explicitly fix the expected values. in this case, I think we're still ignoring Box tests so slightly less urgent, but otherwise this could be breaking tests for other active PRs. I'll go ahead and kick one off now.
also, what changes did you make to the source? we should probably discuss per the "double nested" conversation, since these file storage services are all treated similarly may be worth making sure we're testing consistently.

@ryannikolaidis Shoot! You’re right! Will be more cautious next time.

@rbiseck3 I had them double nested when I built the dropbox and the box connectors. I wanted to have at least 2 levels ov nesting. (there's not much value in having two folders next to each other called nested1 and nested2) I am very curious when/why they got un-nested.
Also there was a partitioned .json file in box that was probably put in when someone was working on the Box destination connector. In our test we were partitioning a partitioned file...
That being said I should not have made those changes in Box.com without consulting with the team. :)


SeedDB Refactoring

Refactoring the seeddb logic to work with promises all over due to all the async behavior.
This lays the ground for further work on adding tests for SeedDB functionality.

@mleanos care to review?

@lirantal I left a line comment. Overall, I like what you've done. Other than what I pointed out, everything is working pretty well.. Tested this with the production env as well.
One thing, that may just be preference, is the use of the callbacks for the promises. Rather than using closures. Is there a specific reason to use one way over the other?
Overall, I really like these changes & how you tied the seeding into the app & mongoose config. That seems like the best way to make this testable and scaleable.

@mleanos  Thanks, I'll address the admin user issue for the production environment state.
What do you suggest we use instead of closures and callbacks? Can you show an example?

@mleanos I updated the PR with the fix for the admin user account.

Check it out, I also added meanjs-core server tests so we can test our own code base.
This is of course just the beginning with the Seed DB feature so there will be more to come.

Closing and re-opening this PR because it seems that Travis CI was not working at all.

@lirantal Travis is having technical issues right now.
https://www.traviscistatus.com/incidents/1r8qwcwdfk90

Ahh, explains it :)

@lirantal I've tested this, and the issue with the remove is fixed. However, in the original version of this seeding, the admin & user accounts were removed & re-created each time when the node env is Development. With these changes, you're not re-creating the admin account if in the Development env. Was that intentional?
In addition to my line comment, I think you can do a couple things differently to reduce the amount of code & keep the logic's SOC in tact...
What about moving the remove functionality outside of the saveUser function? If you burden a new removeUser function with the task of determining whether or not to remove the user (based on env, and admin already existing) then your saveUser can be simplified. Also, by doing so you can reduce the code of your performSeedUser function even further.
For example..
// resolve with success, or reject when there's an error or we don't want to touch, or add, the user
function removeUser(user) {
  return new Promise...
}

And the body of performSeedUser will look something like this..
removeUser(user)
.then(generatePassword)
.then(saveUser(user))
.then(userSeedSuccess)
.then(function () {
  // add your admin user here
})
.catch(reportError);

An alternative to the last call in the above chain, you can refactor performSeedUser to accept a user and call it from start each time you'd like to add a seeded user.
With all that said, I think this is a great start. And I agree that this can be the foundation for a grandeur refactoring of the back-end to make it more testable. How we implement these changes here can set the style for how we refactor, and implement promises, throughout the rest of the back-end. That's part of the reason why I'm so interested in getting this right, or at least understanding the style.

Thanks for the feedback guys.
I'm all about doing things right, so let's work through this PR together to get everything out right as it should be.

@ilanbiala and @mleanos
I've refactored and fixed formatting styles to conform with the comments, please review.
P.S. Going with the closures inside the functions is preferred IMO on moving the user object around.

@lirantal This looks really good! It's much cleaner & easier to follow. And I think it can be extended further in the future if need be.
I left a couple of minor comments. I'm going to spend some time with the tests now. I think we're pretty much there with this. I'm a little concerned that seedTheUser is still dependent on checking the environment & username, but I don't think we have a choice here with this specific use case; and it's a very minor concern for me.
One thing we might want to introduce at this time, is adding seed db options to the environmental configs. For instance, we could add the user info to the config. I can see users wanting to control that specifically based on the environment that the app is running in.
I had an issue with the username being set to "admin" in my development environment. I already had a user with "admin" for the username. The seeding removed this user & I lost context of data that was related to this user. Now might be as good of a time as any to do this, if we can see the need for this to be added to the config.

@ilanbiala @mleanos ready. even increased our coverage by 2%! ;-)
@mleanos about further refactoring to environment variables, how about we work on that in another PR?

@lirantal @ilanbiala @mleanos

I prefer closures as well!
Great job! Looks great. :-D
Let's leave the env variables refactor for another PR.
I think travis is failing on 0.10 because native promises wern't supported until 0.12. If we want to keep 0.10 support, we need to use Q or Bluebird.

Great, thanks Cody.
With regards to versions, v0.10 did receive a backport for Promises support with an update for the rest of the V8 features so we're ok with v0.10 users and no need to deprecate support for that. The build failed for some other reason with db timeouts so I restarted it and it passes just fine now.

@lirantal Good to know. Thanks!
LGTM

@lirantal I left a comment about the empty closure. I agree with your use of the closure there. However, I wanted to clarify why I think we're seeing that behavior.
I'm very pleased with how this turned out! Thanks for working through this with me. I've definitely learned a few things. I think this is ready. LGTM.

@lirantal One thing I forgot to mention.. The tests are logging the seed db events to the console. This is disruptive to the output of the tests. Let's work on that in a separate PR, so we can get this out.

Sure. I noticed that too but didn't have much idea how to further work it out.
You're welcome to submit an enhancement PR with more tests as well.
But hey, we already increased coverage by 2% to 60%, that's awesome! :)

:+1:

@lirantal Once this is merged, I'll work on adding the user info the configs, and adding tests :)
And yea, pretty awesome, we're at 60%! Great job leading the charge on the test coverage!!


PP-1386 - [M&M] - Add dropwizard metrics through graphite UDP reporter

WHAT
A brief description of the pull request:

We added two environment variables for host and port for
metrics.
If the variables are not passed, default values are going to
get picked up

with @simad

👍  LGTM


Existing ~/.eye directory causes eye to misbehave

If the server was rebooted uncleanly, the existing ~/.eye directory causes eye to think processes are running when they are not, or to kill the wrong processes (based on PID?), or just not to start at all.
A quick fix is to rm -$ ~/.eye followed by eye load recorder.eye after a server reboot, but would be nice if eye could detect if it was rebooted?
I'm not sure how eye internals work, but maybe it can check for PID as well as process name or something before assuming a process is up when it is not?

dup of #62


Add default notes for when a deletion procedure is clear

When no note is provided and an email address is provided, display a default note.
When no note is provided and the difficulty is "easy", display a (different) default note.

I'm not sure if the "url" field is required. If it is, I suspect that the vast majority of "easy" entries provide a direct URL to the account/information deletion place.
Closes #653

Per CONTRIBUTING, url is required so this looks fine to me.


Add gomod-doccopy to list of requirements

Actually, do you mind adding a note about installing bazaar and also this under the "Add dependencies" section? I added it to the other repo's README already.

GO111MODULE=on go get github.com/terraform-providers/terraform-provider-xyz (where xyz is the name of the provider)

If this fails with an error like this:

# contrib.go.opencensus.io/exporter/ocagent ../../../../pkg/mod/contrib.go.opencensus.io/exporter/ocagent@v0.4.2/transform_stats_to_metrics.go:210:61: cannot use data.ExemplarsPerBucket (type []*metricdata.Exemplar) as type []*exemplar.Exemplar in argument to bucketsToProtoBucket

..then copy the replace section from this into the go.mod file in this repo. It should look something like this:

replace (
  contrib.go.opencensus.io/exporter/ocagent => contrib.go.opencensus.io/exporter/ocagent v0.4.12
  github.com/Nvveen/Gotty => github.com/ijc25/Gotty v0.0.0-20170406111628-a8b993ba6abd
  github.com/Sirupsen/logrus => github.com/sirupsen/logrus v1.4.2-0.20190403091019-9b3cdde74fbe
  github.com/golang/glog => github.com/pulumi/glog v0.0.0-20180820174630-7eaa6ffb71e4
  github.com/terraform-providers/terraform-provider-azurerm => github.com/pulumi/terraform-provider-azurerm v0.0.0-20190417123607-dd01e8265e07
)

Yes, I’ll open a separate PR for these changes. It’s not quite as cut and dry as copying the replace lines, sadly (though that worked in this case)

Ah ok. Got it.


Requests 2.11.0 and Python 3.2

requests 2.11.0 is incompatible with python 3.2, but is installable:
# pip install requests
Collecting requests
  Downloading requests-2.11.0-py2.py3-none-any.whl (514kB)
    100% |████████████████████████████████| 516kB 1.4MB/s
Installing collected packages: requests
Successfully installed requests-2.11.0
You are using pip version 7.1.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
# python --version
Python 3.2.6

This breaks dependencies for our clients. Could you please make it so 2.11.0 either works properly with python 3.2, or alternatively, make it so it is not installable in Python 3.2?

Hey @ppolewicz, I believe you can find answers addressing this in #3479, primarily @Lukasa's comment here.
pip is currently unable to filter out libraries based on python version, but it was also noted in #3479 that this may be available in the distant future. I'm not positive on when 3.2 was officially deprecated for Requests (it was before 2.11.0) but it was removed from the trove list in setup.py in May 2013. Unfortunately, I don't believe there's any intent to reintroduce support for 3.2, per @sigmavirus24 comments.

I'm going to have to question the comments about 'This isn't possible to prevent installation'
In setup.py if you have something along the lines of the following, if sys.version_info not in [(2,6),(2,7),(3,3),(3,4),(3,5)] than you could raise an error during pip install requests if the python version is outside one of the supported versions

If @Lukasa wants I'd be happy to create a PR for thisfor 2.11.1

Tetra that does not work with universal wheels which is how most users will be experiencing requests installation these days. That only works for source distributions and we are not going to stop distributing wheels.
Sent from my Android device with K-9 Mail. Please excuse my brevity.

This is correct: right now the overwhelming majority of our users download wheels. A quick check of the PyPI download statistics reveals that for the 2.10.0 release last month, the wheel was downloaded 4,087,469 times, and the sdist was downloaded 364,911 times, making the wheels 11.2 times as popular a download as the sdist.
A solution that works only for the sdist is no solution at all.
I should note, as well, that adding a setup.py exception that prevents installation doesn't actually provide a useful behaviour. The most useful behaviour would be if pip clamped the maximum version it would be willing to install on Python 3.2: e.g., if we could say "the last version that supported Python 3.2 was Requests 1.2.3, so go have fun with that". But that doesn't exist either.
I'm open to being told that there is a good way to handle this, but right now there just isn't.

Could we fix it in 2.11.1 by changing those two lines to use [six.u()] (https://pythonhosted.org/six/#six.u)?
I understand that 3.2 is no longer supported by requests, but pip being broken as it is with the version mapping causes trouble for some people. Requests are extremely popular, so it is bound to break something somewhere. I think a minor change of 2 LOC to prevent this would be acceptable?

I understand that 3.2 is no longer supported by requests, but pip being broken as it is with the version mapping causes trouble for some people.

I'm sorry, I don't quite understand what you are referring to @ppolewicz.

I think a minor change of 2 LOC to prevent this would be acceptable?

@ppolewicz except that minor change implies Python 3.2 support. Python 3.2 as a product itself is unsupported. The number of users it has according to PyPI is incredibly small. I don't see a whole lot of value in changing those two lines, but @Lukasa and @kennethreitz may disagree. I'm not against it, I just don't agree that we should be implicitly supporting an unsupported version that is mostly not used (regardless of how easy it is).

@ppolewicz In principle, yes, we could do that. In practice, doing so postpones the inevitable. You'll note that pip itself dropped support for Python 3.2 in 8.0.0. People who refuse to upgrade to Python 3.3, which was released four years ago, are going to need to get comfortable with the idea that their software ecosystem is going to start breaking.

Requests are extremely popular, so it is bound to break something somewhere. I think a minor change of 2 LOC to prevent this would be acceptable?

This logic is just fundamentally flawed. The same argument can be made to support Python 3.0 and Python 3.1, and Python 2.5 and earlier. Each change is only a few LoC, but at a certain point we're just holding ourselves back. 3.2 is no longer receiving security patches from Python dev, which means that it is no longer supported by Python-dev: it should not be used anymore, because it cannot be made to be secure.
I have exactly no interest in supporting Python 3.2 for free. Users who refuse to upgrade to Python 3.3 can apply patches themselves. Users who cannot upgrade to Python 3.3 because their LTS distribution will not ship it to them should contact their distributor, who can either apply appropriate patches themselves or who can pay the Requests development team to continue to support Python 3.2. Those are the two options I will find acceptable.

Fair enough.
Thank you for a detailed explanation.


Bot stopped working

Even after Gen 2 Pokemon, though i was getting error "Unexpected moveset [Rock Throw, Rock Slide]" , Bot was still working. But now bot stopped working ( a couple of hours back i would say ) and now getting this error. Any idea what would have caused this ?

pi@raspberrypi:~/PokemonGo-Bot $ ./run.sh
2017-02-17 21:50:31,608 [       cli] [INFO] PokemonGO Bot v1.0
2017-02-17 21:50:31,629 [       cli] [INFO] commit: 26117e6e
2017-02-17 21:50:31,648 [       cli] [INFO] Configuration initialized
2017-02-17 21:50:31,649 [pokemongo_bot.health_record.bot_event] [INFO] Health check is enabled. For more information:
2017-02-17 21:50:31,650 [pokemongo_bot.health_record.bot_event] [INFO] https://github.com/PokemonGoF/PokemonGo-Bot/tree/dev#analytics
2017-02-17 21:50:31,671 [requests.packages.urllib3.connectionpool] [INFO] Starting new HTTP connection (1): www.google-analytics.com
(4594) wsgi starting up on http://<IP_ADDRESS>:4000
[2017-02-17 21:50:31] [SleepSchedule] [INFO] Next sleep at 12:01:35, for a duration of 05:03:53
[2017-02-17 21:50:31] [PokemonGoBot] [INFO] Setting start location.
[2017-02-17 21:50:31] [PokemonGoBot] [INFO] [x] Coordinates found in passed in location, not geocoding.
[2017-02-17 21:50:31] [PokemonGoBot] [INFO] Location found: 40.765727, -73.976226 (40.765727, -73.976226, 8)
[2017-02-17 21:50:31] [PokemonGoBot] [INFO] Now at (40.765727, -73.976226, 8)
[2017-02-17 21:50:31] [PokemonGoBot] [INFO] Login procedure started.
[2017-02-17 21:50:38] [PokemonGoBot] [INFO] Login successful.
[2017-02-17 21:50:38] [PokemonGoBot] [INFO] Captcha Check Passed
[2017-02-17 21:50:38] [PokemonGoBot] [INFO]
[2017-02-17 21:50:40] [PokemonGoBot] [INFO] [x] Error while opening cached forts: [Errno 2] No such file or directory: u'/home/pi/PokemonGo-Bot/
pokemongo_bot/../data/recent-forts-roxane_13.json'
[2017-02-17 21:50:41] [   Pokemon] [ERROR] Unexpected moveset [Rock Throw, Rock Slide] for #218 Slugma, please update info in pokemon.json and c
reate issue/PR
[2017-02-17 21:50:41] [   Pokemon] [ERROR] Unexpected moveset [Cut, Seed Bomb] for #191 Sunkern, please update info in pokemon.json and create i
ssue/PR
[2017-02-17 21:50:41] [   Pokemon] [ERROR] Unexpected moveset [Rock Smash, Brick Break] for #66 Machop, please update info in pokemon.json and c
reate issue/PR
[2017-02-17 21:50:41] [   Pokemon] [ERROR] Unexpected moveset [Scratch, Aerial Ace] for #190 Aipom, please update info in pokemon.json and creat
e issue/PR
[2017-02-17 21:50:41] [   Pokemon] [ERROR] Unexpected moveset [Tackle, Aerial Ace] for #165 Ledyba, please update info in pokemon.json and creat
e issue/PR
[2017-02-17 21:50:41] [   Pokemon] [ERROR] Unexpected moveset [Bite, Dazzling Gleam] for #209 Snubbull, please update info in pokemon.json and c
reate issue/PR
_inventory was not initialized
_inventory was not initialized
[2017-02-17 21:50:41] [       cli] [INFO]
[2017-02-17 21:50:41] [       cli] [INFO] Ran for 0:00:09
[2017-02-17 21:50:41] [       cli] [INFO] Total XP Earned: 0  Average: 0.00/h
[2017-02-17 21:50:41] [       cli] [INFO] Travelled 0.00km
[2017-02-17 21:50:41] [       cli] [INFO] Visited 0 stops
[2017-02-17 21:50:41] [       cli] [INFO] Encountered 0 pokemon, 0 caught, 0 released, 0 evolved, 0 never seen before ()
[2017-02-17 21:50:41] [       cli] [INFO] Threw 0 pokeballs
[2017-02-17 21:50:41] [       cli] [INFO] Earned 0 Stardust
[2017-02-17 21:50:41] [       cli] [INFO] Hatched eggs 0
[2017-02-17 21:50:41] [       cli] [INFO]
[2017-02-17 21:50:41] [       cli] [INFO] Highest CP Pokemon:
[2017-02-17 21:50:41] [       cli] [INFO] Most Perfect Pokemon:
Traceback (most recent call last):
File "pokecli.py", line 865, in
main()
File "pokecli.py", line 195, in main
bot = start_bot(bot, config)
File "pokecli.py", line 147, in start_bot
bot.start()
File "/home/pi/PokemonGo-Bot/pokemongo_bot/init.py", line 155, in start
init_inventory(self)
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1422, in init_inventory
_inventory = Inventory(bot)
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1266, in init
self.refresh()
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 1276, in refresh
i.refresh(inventory)
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 75, in refresh
self._data = self.retrieve_data(inventory)
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 71, in retrieve_data
ret[key] = self.parse(item)
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 493, in parse
return Pokemon(item)
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 982, in init
self.charged_attack = ChargedAttacks.data_for(data['move_2'])  # type: ChargedAttack
File "/home/pi/PokemonGo-Bot/pokemongo_bot/inventory.py", line 652, in data_for
attack_id, cls.name))
ValueError: Attack 251 not found in ChargedAttacks
[2017-02-17 21:50:41] [sentry.errors] [ERROR] Sentry responded with an error: 'ascii' codec can't decode byte 0x9c in position 1: ordinal not in
range(128) (url: https://app.getsentry.com/api/90254/store/)
Traceback (most recent call last):
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/raven/transport/threaded.py", line 174, in send_sync
super(ThreadedHTTPTransport, self).send(data, headers)
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/raven/transport/http.py", line 47, in send
ca_certs=self.ca_certs,
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/raven/utils/http.py", line 66, in urlopen
return opener.open(url, data, timeout)
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/future/backports/urllib/request.py", line 494, in open
response = self._open(req, data)
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/future/backports/urllib/request.py", line 512, in _open
'_open', req)
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/future/backports/urllib/request.py", line 466, in _call_chain
result = func(*args)
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/raven/utils/http.py", line 46, in https_open
return self.do_open(ValidHTTPSConnection, req)
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/future/backports/urllib/request.py", line 1284, in do_open
h.request(req.get_method(), req.selector, req.data, headers)
File "/usr/lib/python2.7/httplib.py", line 1039, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1073, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 1035, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 877, in send_output
msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9c in position 1: ordinal not in range(128)
[2017-02-17 21:50:41] [sentry.errors.uncaught] [ERROR] [u'ValueError: Attack 251 not found in ChargedAttacks', u'  File "pokecli.py", line 865,
in ', u'  File "pokecli.py", line 195, in main', u'  File "pokecli.py", line 147, in start_bot', u'  File "pokemongo_bot/init.py", l
ine 155, in start', u'  File "pokemongo_bot/inventory.py", line 1422, in init_inventory', u'  File "pokemongo_bot/inventory.py", line 1266, in _
init', u'  File "pokemongo_bot/inventory.py", line 1276, in refresh', u'  File "pokemongo_bot/inventory.py", line 75, in refresh', u'  File "
pokemongo_bot/inventory.py", line 71, in retrieve_data', u'  File "pokemongo_bot/inventory.py", line 493, in parse', u'  File "pokemongo_bot/inv
entory.py", line 982, in init', u'  File "pokemongo_bot/inventory.py", line 652, in data_for']
Fri 17 Feb 21:50:41 UTC 2017 Pokebot  Stopped.
Press any button or wait 20 seconds to continue.
^CTraceback (most recent call last):
File "pokecli.py", line 48, in
from pgoapi.exceptions import NotLoggedInException, ServerSideRequestThrottlingException, ServerBusyOrOfflineException, NoPlayerPositionSetE
xception
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/pgoapi/init.py", line 50, in
from pgoapi.pgoapi import PGoApi
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/pgoapi/pgoapi.py", line 35, in
from pgoapi.rpc_api import RpcApi
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/pgoapi/rpc_api.py", line 45, in
from pgoapi.utilities import to_camel_case, get_time, get_format_time_diff, Rand48, long_to_bytes, f2i
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/pgoapi/utilities.py", line 34, in
from geopy.geocoders import GoogleV3
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/geopy/init.py", line 13, in
from geopy.geocoders import * # pylint: disable=W0401
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/geopy/geocoders/init.py", line 113, in
from geopy.geocoders.ignfrance import IGNFrance
File "/home/pi/PokemonGo-Bot/local/lib/python2.7/site-packages/geopy/geocoders/ignfrance.py", line 5, in
import xml.etree.ElementTree as ET
File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 127, in
from . import ElementPath
File "/usr/lib/python2.7/xml/etree/ElementPath.py", line 62, in
"("
File "/home/pi/PokemonGo-Bot/lib/python2.7/re.py", line 194, in compile
return _compile(pattern, flags)
File "/home/pi/PokemonGo-Bot/lib/python2.7/re.py", line 249, in _compile
p = sre_compile.compile(pattern, flags)
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_compile.py", line 572, in compile
p = sre_parse.parse(p, flags)
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 696, in parse
p = _parse_sub(source, pattern, 0)
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 317, in _parse_sub
itemsappend(_parse(source, state))
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 656, in _parse
p = _parse_sub(source, state)
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 317, in _parse_sub
itemsappend(_parse(source, state))
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 522, in _parse
item = subpattern[-1:]
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 141, in getitem
return SubPattern(self.pattern, self.data[index])
File "/home/pi/PokemonGo-Bot/lib/python2.7/sre_parse.py", line 90, in init
def init(self, pattern, data=None):
KeyboardInterrupt
Fri 17 Feb 21:51:03 UTC 2017 Pokebot  Stopped.
Press any button or wait 20 seconds to continue.

I've seen this too. Are you getting picture captcha? I usually connect from my phone on the same lat lon as the bot and at times, the phone pops up the captcha. I've seen this exact error when that happens. You can usually control-c out of it and start it up again. Just make sure you clear the captcha if that is occurring. Bot is still running for me with the same move errors but still hunting :)

Attack 251, wild charge. New moves.
Download my fast moves and charge moves json files and replace yours in data folder.
I have the files, look for the link in the gen 2 issue thread.

Worked. However still getting "Unexpected moveset" error but bot works

Unexpected moveset is due to information not updated in pokemon.json
If you have the time, you can try to help us. Refer to #5917


Support for openssh

So I was thinking of writing a utility for deploying an application running in docker on a remote system through ssh and came across this crate. It looks to me like this could go well with this crate https://docs.rs/openssh/latest/openssh/index.html#.
What do you think about this approach? Do you think it would be sensible to modify this library to work with both std::process::Command and openssh::process::Command?

Oh. I figured out that docker cli has -H parameter that may be used for connecting to daemon using ssh. Using openssh command does not make sense then as this may be implemented just by using one more argument.


matchSet.addX: hotspot

Hi,
I am doing some performance tests to see If I can use this library instead something I built.
On most of my benchmarks Quamina is faster, there are two main spots which caused terrible degradation:

Flattening - I am passing large objects (7kb~ in size, with nesting) and flattening kills the performance, but since I know the list of fields (& paths) being searched on, I can reduce the incoming event to a smaller one probably.
matchSet.addX - currently it's duplicating the matchSet (even if there are no new matches), this is causing a major degradation.

I did change locally of addX to update in place instead of creating new matchSet, and this is the perf difference I see:
Before:
Benchmark_Quamina-10               27054             49177 ns/op           43311 B/op        160 allocs/op

After:
Benchmark_Quamina-10              221080              6717 ns/op            8932 B/op         28 allocs/op

From what I see, it's done for concurrency concern, but looking at - https://github.com/timbray/quamina/blob/44774840ebb4f07fc0fd5739c4ae4290d74f94ed/core_matcher.go#L151 it looks like there is no concurrent access.
And maybe simple RWMutex can help us instead of duplication?
I can submit a PR to fix this, but trying to understand what are the option we have in hand.

First of all, on the flattening, I suspect it's going to be hard to improve things very much.  The code in flatten_json that skips over unwanted fields is pretty fast so it may not be cost-effective to write code to remove events before testing them.
I think the best approach to speeding up flattening would be to use protobufs or Avro or something where there are pointers to the fields so you don't have to scan over all the bytes that aren't interesting.  Several people have told me that they think in principle it should be straightforward to write such a hyper-efficient flattener, but AFAIK nobody has done it yet.

Oh, and I should say, thanks for the input!
On the concurrency stuff… that's surprising.  Could I ask a favor? Go into concurrency_test.go and try increasing the intensity by increasing the "n" and "tasks" variables, with your update-in-place change?
The concurrency we're worried about is between AddPattern() and MatchesForEvent(). There can only be one AddPattern thread but there can be many MatchesForEvent. First symptom was one of the MatchesForEvent coroutines reading a map[] while an AddPattern goroutine was updating it, the runtime throws a specific panic. Second symptom: Same, only for a slice, except this eventually causes an invalid-index panic.  The second symptom was much rarer and harder to reproduce.
The update-by-replace in matchSet and other places fixed the first problem, and then the atomic.Value in coreMatcher, fieldMatcher, and valueMatcher fixed the second.   Possibly the atomic.Value also removed the need for the fancy matchSet?  That doesn't seem true in my mind, but I could easily be wrong.
I wonder if the case where matchSet.addX() is adding no new values is easy to detect and simply return the input?  I'll have a look at that.
BUT now it dawns on me that the matchSet that is used only in the AddPattern thread, which is the variable match in matchesForSortedFields() is accessed only in that thread and I'm sure it could safely use update-in-place.  If you wanted to create a new type for that purpose, I'm pretty sure I'd take that PR as a first step.

Hi,
Thanks for the responses! I will give them a detailed look & answers tomorrow (or the day after).

I wonder if the case where matchSet.addX() is adding no new values is easy to detect and simply return the input? I'll have a look at that.

From my data, before:
Quamina-10               26460             47247 ns/op           43318 B/op        160 allocs/op

After adding len check on exes in addX:
Quamina-10               35757             33191 ns/op           30393 B/op        129 allocs/op

It gives a good improvement, but it's not the one I need in order to replace my existing implementation.

If you wanted to create a new type for that purpose, I'm pretty sure I'd take that PR as a first step.

You are suggesting a new type which the same as matchSet but does update-in-place? it sounds ok, but what do you think about two methods / one method ?
// one method option
addX(updateInPlace bool, exes ...X) *matchSet

// two methods option
addX(exes ...X) *matchSet
unsafeAddX(exes ...X) *matchSet // does update-in-place

About flatenning, I admit I haven't looked too much at the code what it returns and how does it works, but in my solution I have am getting those large events and I managing to work with them really efficiently.
I achieve that by using jsoniter / jx which allows me to read subset of the event, and instead of flattening I am doing lazy-reads against that JSON.
What helped me a lot here, is to re-strucutre my object, so instead of:
{
    field1: "a", // have match against
    field2: "b", // have match against
    ... rest of the 7kb data ..
}

I am doing:
{
     "context": { field1: "a", field2: "b" },
     "payload": { ... 7kb data .. }
}

With this approach jsoniter/jx don't need to traverse all of the properties, just get to context property and that's it.
I'll give flattening a deeper look and see why this re-structuring don't help Quamina, since essentially the flattener should skip the whole "payload" object.
Again, thanks for your feedback and for this awesome library!

Haven't seen your code but I assume you mean something like
  if len(exes) == 0 {
    return m
  }

then yes, please include that in any PR.
Yes, two methods on the existing matchSet class is probably better. Maybe addXSingleThreaded() or some such?
Don't know jsoniter or jx but I'm super-interested in anything that could make flattening faster because last time I profiled it was still >50% of matching latency.  So I look forward to hearing what you discover.

Submitted PR for matchSet changes - https://github.com/timbray/quamina/pull/109, by the way I see there are no formatting enforced with gofmt - would you accept a PR for doing gofmt on the files and ensuring that on CI?
Regarding the flattener, I have looked into it, and given this pattern:
{
	"context": {
		"field1": ["a"],
		"field2": ["b"]
	}
}

Event:
{
     "context": { "field1": "a", "field2": "b" },
     "payload": { ... 7kb data .. }
}

It looks like the flattener goes into "payload" and lookup for properties called "context" / "field1" / "field2", which makes it not very efficient for my case (and maybe others as well), instead what I suggest is to:

Keep track on "paths" for patterns - in my case it's context.field1 / context.field2
In flattening phase, pluck only those paths and ignore all others.

Good catch on gofmt. I have my IDE wired up for gofmt-on-save and my bad for not noticing the miss.  Yess, I'd love a CI PR for that. But I thought we already had something - @embano1 am I wrong on that?
Yeah, at the current time the flattener traverses the event end-to-end to be sure it hasn't missed any relevant fields. Which gives me an idea: enhance the NameTracker interface to give a list of all the fields in play and then the Flattener could know when it's found everything and stop looking. I'd be a little worried that for short simple events this could actually slow things down, so it might make sense to have a heuristic and only do this check for events larger than a certain threshold.

Good catch on gofmt. I have my IDE wired up for gofmt-on-save and my bad for not noticing the miss. Yess, I'd love a CI PR for that. But I thought we already had something - @embano1 am I wrong on that?

It's because the linter wasn't enabled in golangci, I have enabled it here - https://github.com/timbray/quamina/pull/110 and fixed all linting errors.

enhance the NameTracker interface to give a list of all the fields in play and then the Flattener could know when it's found everything and stop looking.

This exactly what I have in mind, how do you think to implement such an API? it means we need to store list of pattern fields in a coreMatcher and pass them via NameTracker interface. The problems come in hand is how to handle deletions, because then we need some kind of ref-count which makes thing complex.
I'll try to implement a POC of external flattener which accepts a list of paths and will do flattening using jx / jsoniter I'll check small objects (where patterns covers 90% of properties) and large objects and will report back with code-sample and my findings.

Some quick update..
I wrote another flattener based on Jx, which accepts "paths" (tree based paths needed pluck) -
goos: darwin
goarch: arm64
pkg: github.com/yosiat/quamina-flatenner
Benchmark_Quamina_JxFlattener-10                  143412              7351 ns/op              24 B/op          3 allocs/op
Benchmark_Quamina_Flattener-10                     83698             14388 ns/op             360 B/op         23 allocs/op
Benchmark_LargePayload_QuaminaFlattner-10           77971             15326 ns/op            1240 B/op         44 allocs/op
Benchmark_LargePayload_JxFlattner-10               145306              8318 ns/op             904 B/op         24 allocs/op
PASS
ok      github.com/yosiat/quamina-flatenner     5.347s

Tomorrow I'll run Quamina flattener tests against it and make sure it's compliant and then I'll push the code to github (to a repo) so you can give a look at it.

What is the first column in the benchmark output?  Ideally we'd like to run some of the existing Citylots benchmarks replacing json_flattener with yours and compare the performance.
Just one caveat: I am very very reluctant to add dependencies to Quamina, because I see this as a very low-level and horizontal library.

What is the first column in the benchmark output?

It's the standard output of go benchmarks, it's the number of loops.

Ideally we'd like to run some of the existing Citylots benchmarks replacing json_flattener with yours and compare the performance.

Where they exists? I can run them and publish results here until I publish the full code.

Just one caveat: I am very very reluctant to add dependencies to Quamina, because I see this as a very low-level and horizontal library.

Totally makes sense and acceptable, I used Jx since it was faster and easier to get to a result which shows my point, I assume the changes I did can be done with encoding/json, it was too complex for me to change existing code to get to my result.
Once I publish my code, I believe you will understand easily what I did and how I improved the performance and what you can do in order to adjust existing code for it.

Have a look at benchmarks_test.go.  Unfortunately I previously didn't know about the built-in Go benchmarking support so they don't yet take advantage of that.

Cool, I looked at it and I'll need to do some adjustments to my code to make it work.
Going to sleep, will do it tomorrow ~

This exactly what I have in mind, how do you think to implement such an API? it means we need to store list of pattern fields in a coreMatcher and pass them via NameTracker interface. The problems come in hand is how to handle deletions, because then we need some kind of ref-count which makes thing complex.

The first/easiest thing I thought of would be to have an API in `NameTracker like
func (nt *NameTracker) GetFieldSet() map[string]bool

The idea is it would return a "set" of all the field names that are used and, for convenience, the set would be writeable, so whenever you encounter one in the element you remove it from the set and as soon as the set is empty you stop parsing the event.  It would be a bit expensive to generate but you're only going to be using this API on big events, so maybe OK?  But I didn't think a lot, quite likely you have a better idea.
BTW, I think your technique of moving the interesting fields up to the front of the event is very clever, but probably not a reasonable thing to ask users to do in the general case.  But this new API might be a good idea anyhow.

Hi,
Ran the benchmarks_test.go I saw only one benchmark for BigShellStyle -
# Quamina
Field matchers: 27 (avg size 1.000, max 1), Value matchers: 1, SmallTables 54 (avg size 15.500, max 28), singletons 0
428,547.72 matches/second with letter patterns

# Jx
Field matchers: 27 (avg size 1.000, max 1), Value matchers: 1, SmallTables 54 (avg size 15.500, max 28), singletons 0
1,367,947.02 matches/second with letter patterns

I have published the code up here - https://github.com/yosiat/quamina-flatenner
I made some hacks in Quamina to expose some internal methods in order to run the benchmarks externally, so it will be be a bit hacky to run it locally, but what's needed is:

Clone the quamina-flatenner project
Install https://github.com/rogpeppe/gohack
Run go build and then gohack get -vcs -f github.com/timbray/quamina
In the gohack folder, checkout this branch - https://github.com/yosiat/quamina/tree/flat-hack, flat-hack.
And then simply go test -v

I'll make this flow easier later today, instead of external repo I'll put inside a fork branch.
Some notes:

In my "flat-hack" branch, I added used paths map, same as namesUsed in order to get list of paths passed to flatenner.
Haven't check the full correctness of my flatenner - the benchamrks_test.go and my little tests are passing, but hopefully soon in a fork branch I'll be able to get existing tests running (and then passing ;) )
The benchmarks I posted above, currently I can't share because they contain internal data, I'll anonymise it and will post the benchmarks as well.

A bit about the source:

https://github.com/yosiat/quamina-flatenner/blob/main/paths.go - Is a tree structure for indexing the paths, so I can traverse along while traversing objects in flatenner.
https://github.com/yosiat/quamina-flatenner/blob/main/jx.go - Is the actual flatenner which uses jx, I copied some methods from Quamina mostly to help with ArrayTrail logic.

Update:
I pushed the sources to my side branch, so it will be easier to use.

Checkout  https://github.com/yosiat/quamina/tree/flat-hack
Run go build (to install jx)
To run benchmarks, I prefixed them with Test_JX - so go test -v -run "^Test_JX" will suffice.

Oops… the benchmark we most care about isn't in benchmarks_test (oops, sorry) it's TestCityLots in quamina_test.go. Very strongly dependent on flattener performance.
I have to go do other stuff for a while, will get back to this.

Ok, now it's getting a smaller difference -
=== RUN   TestCityLots
Field matchers: 7 (avg size 1.500, max 3), Value matchers: 6, SmallTables 0 (avg size n/a, max 0), singletons 6

173,434.09 matches/second

--- PASS: TestCityLots (1.64s)
=== RUN   TestCityLots_JX
Field matchers: 7 (avg size 1.500, max 3), Value matchers: 6, SmallTables 0 (avg size n/a, max 0), singletons 6

177,609.63 matches/second

And the profiles are the same, most of the time is on storeArrayElementField.

Yeah, the Patterns applied in that benchmark force the flattener to process the whole record, which includes some large-ish arrays of floating-point numbers, so it's pretty well a worst-case.  But actually, there are two distinct problems:

Making the flattener efficient when processing lots of fields
Making the flattener smarter about skipping fields it doesn't need to look at (what you've been working on mostly)

OK, so once again, thanks for this work - once we've landed this PR I'll add you as a project contributor. Now, a confession: You've been doing so much work that I've sort of lost track of which PRs I should review and when. Feel free to send me a message, email or Signal<PHONE_NUMBER>), when you'd like me to take a close look at something.

OK, so once again, thanks for this work - once we've landed this PR I'll add you as a project contributor.

Thanks a lot!

I've sort of lost track of which PRs I should review and when

In terms of PRs, we have:

https://github.com/timbray/quamina/pull/112 - fixing linter
https://github.com/timbray/quamina/pull/109 - matchset.addX performance, once the linter fixing is merged I'll rebase and adapt

And then we need to decide on how to proceed with the flattener changes: My approach is clear - track list of paths used and flatten only them, but I am not sure how to proceed here:

Tracking list of paths is easy when we are considering additions only, but when we consider removals (of patterns) is becomes complex, because then we need to make sure to remove a path only when it's not used by all patterns. I haven't looked at how deletions work currently, so I have no idea on this.
My flattener uses Jx and you said that you don't want an external dependency which makes sense, doing this with existing flattener will be complex, I can try to make it work but it will make some time to get my around existing parser.

So my work around flattener will allow me to use an external one (which is my one) to continue conducting my tests in the internal project and see the e2e performance of it. But I am not sure it's ready for something that we can Merge/Review.

Feel free to send me a message, email or Signal.

Thanks! I'll keep the discussions here in the open in GitHub and if case an arise I'll send you an email (I have sent you an email, so you will have my private as one).

We can probably close this issue, but maybe start another one on flattening, because there is useful discussion in here. I just had another idea: Instead of making NameTracker smarter, we could consider a Flattener implementation that took config options, for example "when you come to this field you can stop, nothing useful after it".  Because it sounds like skipping parts of the event is only really useful when you have hand-crafted events and the caller has inside knowledge about the structure.

Closing this issue and suggested opened new for flattening - https://github.com/timbray/quamina/issues/113.


Add a page for unviewed BLAST submissions

Description
When a BLAST form is submitted, open a page for unviewed BLAST submissions.
Might use the ButtonLink component from https://github.com/Ensembl/ensembl-client/pull/742 if it is approved in time.
Related JIRA Issue(s)
https://www.ebi.ac.uk/panda/jira/browse/ENSWBSITES-1560
Deployment URL(s)
http://add-unviewed-results-page.review.ensembl.org/blast/unviewed-submissions

@azangru i have approved the code changes but one thing that i have noticed and not sure if it is part of this PR or should be a separate ticket is:

Go to http://add-unviewed-results-page.review.ensembl.org/blast and fill in sequence and select species
click Run and you get send to the unviewed jobs which is fine
Now when you click new job button, it sends you back to the blast form but the form is still filled with the last job that i ran, should i not clear itself?


JSON api description: compound types

Our current JSON api description implementation is kept very minimal and can only support the Rust primitives bool, u{8,16,32,64,128} and i{8,16,32,64,128} as well as the SRML primitives Address and Balance.
To allows for custom user types and tuples we need to extend the set of available types to include compound types. These can be named and unnamed optionally. A compound type is transparent in that it tells how it is structured inside to make it work language agnostic.
{
    "name": "Vec3",
    "fields": [
        { "x": "i32" },
        { "y": "i32" },
        { "z": "i32" }
    ]
}

The above example would demonstrate something like a
struct Vec3 {
    "x": i32,
    "y": i32,
    "z": i32,
}

We also want to support optionally unnamed tuples like the following:
{
    "name": null,
    "fields": [
        "i32", "bool", "Address"
    ]
}

This represents something like the following Rust tuple: (i32, bool, Address)
When name is not null but for instance "Foo" it also represents a tuple struct, like:
struct Foo(i32, bool, Address);

It is important to note that this representation is recursive. So we could easily define something like this:
{
    "name": "Song",
    "fields": [
        { "duration": "u32" },
        { "title": "[u8; 32]" },
        { "date":
            {
                "name": "Date",
                "fields": [
                    { "year": "u16" },
                    { "month": "u8" },
                    { "day": "u8" }
                ]
            }
        }
    ]
}

Implemented by https://github.com/Robbepop/pdsl/pull/41.
Closed.

I have implemented something similar to this for substrate metadata and was hoping it could be used for smart contracts as well. I wonder if any of my code could help? https://github.com/paritytech/substrate/pull/1328
For this particular issue, there are few catches to the proposed solution:

Circular types: It simplify cannot handle circular types without some kind of references. One example is enum LinkedList { Node(T, Box<LinkedList>), Empty }
Duplicated information. Example: struct A { field: u8, field2: u8 }; struct B { field: A, field2: A }. You don't want to store the data for struct A twice.

We might want to consider using a similar or your approach (and code) in the future. Thank you for linking it!

Implemented by https://github.com/paritytech/ink/pull/104
Can be closed once merged.

Merged. Closed.


[BUG] Bot Unable to detect flat size

Describe the bug
Bot ignores the apartment as users set limit is something greater than 0. Bot detects the apartment size as 0 m2
To Reproduce
Run the bot when apartment is available
Screenshots

Environment
MAC OS - 14.4.1
Additional context
[29.04.2024 - 16:17] [main.py:76] [INFO] Checking for internet connection 🔎
[29.04.2024 - 16:17] [main.py:84] [SUCCESS] Online 🟢
[29.04.2024 - 16:17] [io_operations.py:24] [INFO] Loading WBM config ⚙
[29.04.2024 - 16:17] [main.py:107] [INFO] Connecting to 'https://www.wbm.de/wohnungen-berlin/angebote/' 🔗
[29.04.2024 - 16:17] [webDriverOperations.py:252] [SUCCESS] Cookies have been accepted 🍪
[29.04.2024 - 16:17] [webDriverOperations.py:380] [INFO] Looking for flats 👀
[29.04.2024 - 16:17] [webDriverOperations.py:387] [SUCCESS] Found 2 flat(s) in total 💡
[29.04.2024 - 16:17] [webDriverOperations.py:438] [WARNING] Ignoring flat '3-ZIMMER WOHNUNG IN MITTE WBS 140' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:17] [webDriverOperations.py:438] [WARNING] Ignoring flat '2-ZIMMER-WOHNUNG IN MITTE WBS 160/180/220' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:17] [webDriverOperations.py:61] [ERROR] Failed to switch page, last page reached ❌
[29.04.2024 - 16:18] [webDriverOperations.py:504] [INFO] Reloading main page 🔄
[29.04.2024 - 16:18] [webDriverOperations.py:380] [INFO] Looking for flats 👀
[29.04.2024 - 16:18] [webDriverOperations.py:387] [SUCCESS] Found 2 flat(s) in total 💡
[29.04.2024 - 16:18] [webDriverOperations.py:438] [WARNING] Ignoring flat '3-ZIMMER WOHNUNG IN MITTE WBS 140' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:18] [webDriverOperations.py:438] [WARNING] Ignoring flat '2-ZIMMER-WOHNUNG IN MITTE WBS 160/180/220' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:18] [webDriverOperations.py:61] [ERROR] Failed to switch page, last page reached ❌
[29.04.2024 - 16:19] [webDriverOperations.py:504] [INFO] Reloading main page 🔄
[29.04.2024 - 16:19] [webDriverOperations.py:380] [INFO] Looking for flats 👀
[29.04.2024 - 16:19] [webDriverOperations.py:387] [SUCCESS] Found 2 flat(s) in total 💡
[29.04.2024 - 16:19] [webDriverOperations.py:438] [WARNING] Ignoring flat '3-ZIMMER WOHNUNG IN MITTE WBS 140' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:19] [webDriverOperations.py:438] [WARNING] Ignoring flat '2-ZIMMER-WOHNUNG IN MITTE WBS 160/180/220' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:20] [webDriverOperations.py:61] [ERROR] Failed to switch page, last page reached ❌
[29.04.2024 - 16:21] [webDriverOperations.py:504] [INFO] Reloading main page 🔄
[29.04.2024 - 16:21] [webDriverOperations.py:380] [INFO] Looking for flats 👀
[29.04.2024 - 16:21] [webDriverOperations.py:387] [SUCCESS] Found 2 flat(s) in total 💡
[29.04.2024 - 16:21] [webDriverOperations.py:438] [WARNING] Ignoring flat '3-ZIMMER WOHNUNG IN MITTE WBS 140' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:21] [webDriverOperations.py:438] [WARNING] Ignoring flat '2-ZIMMER-WOHNUNG IN MITTE WBS 160/180/220' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:21] [webDriverOperations.py:61] [ERROR] Failed to switch page, last page reached ❌
[29.04.2024 - 16:22] [webDriverOperations.py:504] [INFO] Reloading main page 🔄
[29.04.2024 - 16:22] [webDriverOperations.py:380] [INFO] Looking for flats 👀
[29.04.2024 - 16:22] [webDriverOperations.py:387] [SUCCESS] Found 2 flat(s) in total 💡
[29.04.2024 - 16:22] [webDriverOperations.py:438] [WARNING] Ignoring flat '3-ZIMMER WOHNUNG IN MITTE WBS 140' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:22] [webDriverOperations.py:438] [WARNING] Ignoring flat '2-ZIMMER-WOHNUNG IN MITTE WBS 160/180/220' because the size doesn't match our criteria --> Flat Size: 0.0 m² | User wants it above: 40 m² 🙈
[29.04.2024 - 16:22] [webDriverOperations.py:61] [ERROR] Failed to switch page, last page reached ❌
^CTraceback (most recent call last):
File "/Users/UsmanTatla/Desktop/wbmbot_v2-main/wbmbot_v2/main.py", line 134, in
main()
File "/Users/UsmanTatla/Desktop/wbmbot_v2-main/wbmbot_v2/main.py", line 111, in main
webDriverOperations.process_flats(
File "/Users/UsmanTatla/Desktop/wbmbot_v2-main/wbmbot_v2/helpers/webDriverOperations.py", line 500, in process_flats
time.sleep(int(refresh_internal) * 60)
KeyboardInterrupt
UsmanTatla@MacBookPro wbmbot_v2-main %

@iTatla1 Thanks for reporting this issue.
I've encountered this several times before, and usually it is a problem from the WBM listing itself.
The "Apartment Description list" array usually has about 12 values, but sometimes the person who publishes the listing forgets to mention one of the entries so the array indexes are messed up, and the bot has a certain order of the indexes, when not properly found, things that there is no size, thus the 0.0m2
I can try to make a workaround for it once I have some free time!
Thanks again :) And goodluck in your hunt!


README math not rendering nicely

the doc readme LaTeX doesn't render properly
Was wondering if the readme could be linked in as a jupyter notebook (which does) or implement a solution like this
https://gist.github.com/a-rodin/fef3f543412d6e1ec5b6cf55bf197d7b
to make readability easier

the doc readme LaTeX doesn't render properly
Was wondering if the readme could be linked in as a jupyter notebook (which does) or implement a solution like this
https://gist.github.com/a-rodin/fef3f543412d6e1ec5b6cf55bf197d7b
to make readability easier


Modal Popups Functionality

We have the ability to do Modal Experience Fragments within AEM along with some options. These options include:

Which Fragment to use/show in the modal
ID (used for user initiated clicks to show modal)
Lazy Load Modal Content
Modal Size (Small, Medium, Large (default), Extra-Large)
Trigger type (None - user click, Scroll Percentage, Page Time, Exit Intent)
Scroll percentage value
Page time (in seconds)
Exit intent delay (in seconds)

See below screenshot with our current options/set up.

It also needs to be able to connect with Adobe Analytics for tracking how many views, how many clicks, how many closes, etc...

Some additional context:
Frequency

Show once per visitor.
Show on every visit.
Show only on visit
Show every visits.
Don't show on first visit, show on all return visits

@jindaliiita @jalagari @ravise5 where are we with the Modal functionality?

@rwcsrichardspencer Can you review the URLs in the above PR in terms of functionality?


Persist auth tokens on db

Hello. Is it possible to authorise users by tokens (i.e. generated uuid), that stored on DB (dedicated sessions collection or "tokens" field on users collection)? Without JWT, that stores only on client-side.
Pros: You can invalidate user session on server-side (i.e. if user changed password we need to logout him) or from another session ("Logout from other devices" feature)
Cons: Need to check token on every request.

Token Blacklisting functionality is being tracked in #73 there is some discussion there but currently you'd still have to implement that yourself manually (with a custom JWT verifier).