Why does a media driver restart require that all client processes also restart? [question]

Hello,
In issue #274 , you state:

For Aeron media driver shutdown, the best way to handle that is to restart the Aeron instances and that means restarting the Publications and Subscriptions they use as well. Consider it a hard error and a total restart. The Aeron errorHandler will deliver errors for media driver liveness, etc. so the application does not have to.

Why is a media driver restart so 'dramatic' (for lack of a better word)?
I would like to toy with the concept for having clients that interact via the external media driver "recover". Basically if another JVM that connects to the media driver loses that connection, it would close all publications/subscriptions, close the connection with Aeron.close(), then re-establish a new one by periodically calling Aeron.connect(ctx).
Is it really impossible to survive a media driver restart and what is the reason behind that?

So, I did a test in Windows as follows:

A client is in a loop trying to connect.
When connected the client creates a publication.
Client then enters a loop waiting for a subscriber to appear.

I noticed that the following sequence causes the driver to not be able to run:

Start test client
Wait to see it try to connect a few times unsuccessfully...

Start media driver
Wait to see client manage to connect; it will create the publication and enter the inner loop waiting for a subscriber to arrive.

Kill media driver
Wait for client to notice. It closes the publication and connection context.

Start media driver again
This fails!

package playground;

import java.util.concurrent.atomic.AtomicBoolean;

import org.agrona.CloseHelper;

import io.aeron.Aeron;
import io.aeron.Publication;

public class RunPublisher {
    private final AtomicBoolean failed = new AtomicBoolean(false);

    private final String uri;
    private final int streamId;

    public RunPublisher(String uri, int streamId) {
        this.uri = uri;
        this.streamId = streamId;
    }

    public static void main(String[] args) throws Exception {
        new RunPublisher(args[0], Integer.parseInt(args[1])).run();
    }

    private void run() {
        final Aeron.Context ctx = new Aeron.Context();

        ctx.errorHandler((Throwable error) -> {
            error.printStackTrace(System.out);
            System.out.println("Connection died... Will try to re-establish...");
            failed.set(true);
        });

        Aeron aeron = null;
        Publication publication = null;

        while (true) {
            aeron = null;
            publication = null;

            try {
                System.out.println("Reconnecting...");
                failed.set(false);
                aeron = Aeron.connect(ctx);
                try {
                    System.out.println("Connect succeeded! Add publication...");
                    publication = aeron.addPublication(uri, streamId);
                    try {
                        System.out.println("publication added! Wait for subscriber...");
                        while (!failed.get() && !publication.isConnected()) {
                            System.out.println("No active subscribers detected");
                            Thread.sleep(1000);
                            continue;
                        }
                        if (!failed.get()) {
                            System.out.println("Subscriber arrived! Goodbye!");
                            break;// success!
                        }
                    } catch (Exception e) {
                        e.printStackTrace(System.out);
                        System.out.println("Error waiting for subscriber... Resetting...");
                        continue;
                    }
                } catch (Exception e) {
                    e.printStackTrace(System.out);
                    System.out.println("Add publication failed... Resetting...");
                } finally {
                    System.out.println("close(publication)");
                    CloseHelper.quietClose(publication);
                }
            } catch (Exception e) {
                e.printStackTrace(System.out);
                System.out.println("Connect failed... Will retry...");
            } finally {
                System.out.println("close(aeron)");
                CloseHelper.quietClose(aeron);
            }
        }
    }
}

Apparently when the driver dies, a file handle is left open for the publication on the CLIENT side. This prevents the driver from being able to restart. The exception I get is:
WARNING: C:\Users\karypid\AppData\Local\Temp\aeron-karypid already exists.
INFO: Aeron CnC file C:\Users\karypid\AppData\Local\Temp\aeron-karypid\cnc.dat exists
INFO: Aeron toDriver consumer heartbeat is 30035ms old
Exception in thread "main" java.nio.file.AccessDeniedException: C:\Users\karypid\AppData\Local\Temp\aeron-karypid\publications\UDP-00000000-0-7f000001-40123-cee8cf5a-a-1.logbuffer
	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:83)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
	at java.nio.file.Files.delete(Files.java:1126)
	at org.agrona.IoUtil.delete(IoUtil.java:137)
	at org.agrona.IoUtil.delete(IoUtil.java:128)
	at org.agrona.IoUtil.delete(IoUtil.java:128)
	at io.aeron.CommonContext.deleteAeronDirectory(CommonContext.java:318)
	at io.aeron.driver.MediaDriver.ensureDirectoryIsRecreated(MediaDriver.java:424)
	at io.aeron.driver.MediaDriver.<init>(MediaDriver.java:163)
	at io.aeron.driver.MediaDriver.launch(MediaDriver.java:285)
	at io.aeron.driver.MediaDriver.launch(MediaDriver.java:274)
	at io.aeron.driver.MediaDriver.main(MediaDriver.java:144)

I will try in Linux and post back, but it seems to me that a handle is left lingering...

Try closing all clients, i.e. Aeron.close() before restarting the driver.

The problem is in class Publication I think:
    void releasePublication(final Publication publication)
    {
        verifyActive();

        if (publication == activePublications.remove(publication.channel(), publication.streamId()))
        {
            lingerResource(publication.managedResource());
            awaitResponse(driverProxy.removePublication(publication.registrationId()), null);
        }
    }

verifyActive() throws an exception and publication.channel() is never touched...
Disclaimer: I literally just started looking into this code; I may be way off...

You should just ignore closing the publications and subscriptions and just close the Aeron client. This is more forceful. I'll update the release as it does not need the driver check.

Thank you, it works after commenting out:
                    // System.out.println("close(publication)");
                    // CloseHelper.quietClose(publication);

The change for IllegalStateException should steer users in the right direction. I will add actual subscribers and continue testing (also on Linux) and report here if I run into any other issues.

Related question(s):

Driver timeout milliseconds.

The description reads "The timeout in milliseconds after which the driver is considered dead if it does not update its C'n'C timestamp." In my testing I lowered this value in order to speed up detection of a "dead driver".

Are there any guidelines on its value? 10 seconds seems quite high. How does this affect Aeron if set to (say) 1 second?
Obviously the timeout needs to be set to the same value in all processes using this driver. Unfortunately there is there no system property for it? It's a pity one has to use a custom launcher to set it on the driver side. Would you consider adding one? (I could create a pull request, though it's ridiculously simple).

Ensuring same driver incarnation

I noticed that a new driver will wait until driver timeout milliseconds before trying to take over an existing directory (which makes absolute sense). Still, is there something in Aeron's protocol that discriminates between instances of the driver (to ensure no data is intermixed between an old and new incarnation)? I added a "sessionId" in the application layer protocol that increases monotonically every time I connect to a new driver. This ensures that receiving clients can detect a "new" incarnation sending data. I suspect there's already something available to do this but is it accessible to subscribers/publishers via the API?

I've added the ability to set the driver timeout in milliseconds via the aeron.driver.timeout system property.
Due to garbage collection, page cache flushing, zone reclaims, scheduling starvation, etc. it is unwise to set a timeout less than a few seconds unless you have a very well tuned and setup machine.

The CnC file contains the client liveness timeout that the driver will use. This needs to be communicated to the clients. The timeout that the clients should use for the driver depends on how often the driver updates the timestamp.
 Each time the driver starts it starts with a clean state. If a client knows it is communicating with an "old" driver, it will make assumptions that can not hold. There is no real value to making assumptions like this.

@mjpt777 thanks for this change. Your points about GC/OS pauses is noted.

@tmontgomery That was precisely what I wanted to ensure: that all previous data "from the wire" is discarded. I was worried the new media driver may deliver messages the previous instance received, so therefore the application code needs to defend against it. Based on your explanation I see this is not a concern.

Unfortunately reconnecting in Linux seems to be unstable. Basically, when I restart the media driver, the publisher connects and then tries to add the publication, but I sometimes I get a SIGSEGV.
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f7721a879ef, pid=26194, tid=140149143889664
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 6135 C2 io.aeron.ClientConductor.doWork()I (59 bytes) @ 0x00007f7721a879ef [0x00007f7721a87860+0x18f]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /server/hs_err_pid26194.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

Do you have a line number for the SIGSEGV in the core dump?

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
I've turned that on and will chase it. Unfortunately it's just hard to reproduce.... Only got it twice in the past 8 hours. Will have more time to fiddle so I'm sure I'll manage to track it.

Just a quick update - it seems to me that the SIGSEGV is not the result of some native code but a direct buffer access:
Stack: [0x00007f75e8037000,0x00007f75e8138000],  sp=0x00007f75e8136290,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 5767 C1 org.agrona.concurrent.ringbuffer.ManyToOneRingBuffer.write(ILorg/agrona/DirectBuffer;II)Z (110 bytes) @ 0x00007f76312bba11 [0x00007f76312bb6a0+0x371]
j  io.aeron.DriverProxy.addPublication(Ljava/lang/String;I)J+49
j  io.aeron.ClientConductor.addPublication(Ljava/lang/String;I)Lio/aeron/Publication;+26
j  io.aeron.Aeron.addPublication(Ljava/lang/String;I)Lio/aeron/Publication;+15
j  playground.aeron.AeronConnectionManagerImpl.addPublication(Ljava/lang/String;I)Lio/aeron/Publication;+22
j  playground.aeron.FxPriceAeronPublisher.onConnected()V+42
j  playground.aeron.AeronConnectionManagerImpl.notifyOnConnected()V+16
j  playground.aeron.AeronConnectionManagerImpl.fireOnConnected(Lio/aeron/Aeron;)V+32
j  playground.aeron.AeronConnectionManagerImpl.run()V+121
j  playground.aeron.AeronConnectionManagerImpl$$Lambda$32.run()V+4
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x68c616]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x68cb21]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x68cfc7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x723d80]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0xa69dcf]  JavaThread::thread_main_inner()+0xdf
V  [libjvm.so+0xa69efc]  JavaThread::run()+0x11c
V  [libjvm.so+0x91d9d8]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x7aa1]  start_thread+0xd1

I am going to switch back to my test publisher and see if I can reproduce this with a snippet.
I am also considering a different approach: have a custom media driver launcher that picks a new aeronDirectory every time (e.g. /dev/shm/aeron-${pid}) so that a fresh location is used. Clients that notice the driver heartbeat timed out will call Aeron.close() and open a connection to the new directory. I can have the custom launcher lock a file (e.g. /dev/shm/current) and overwrite the content with the name of the new location, so that clients know where to try and reconnect (as soon as content changes).

If you try with the latest build it should close the client on a driver timeout.

I'm going to close this for now. Any follow up can be opened as specific issues.


Update development-environment.md

This pull request contains changes to improve clarity, correctness and structure.
Description correction:
Corrected installs to install
Please review the changes and let me know if any additional changes are needed.

thank @zeevick10 pls format PR and commit using the conventional commits convention


functions for quicker access to metabolism model inputs

Coverage decreased (-0.2%) to 10.032% when pulling b30d5e2fc29c64368bcc5af4225d167fc8679af3 on aappling-usgs:develop into ac99c373af4006475d321762a70810e7090769d2 on USGS-R:develop.


Support APIs for Releases

This block of APIs:
https://hub.allspice.io/api/swagger#/repository/repoListReleases
Should also support downloading tarballs and attachments.

Closed by #73


POSIX compliance of mc_rtc

Hi,
We are planning to run mc_rtc into a non-Linux or non-Unix, but POSIX compliant OS. For this, I was wondering if mc_rtc is POSIX compliant, such that we could port it into this OS.
Thank you

Hi @magnusmusngi
I think it should be ok if you can get all of the dependencies of mc_rtc in this environment and if it has a sufficiently recent compiler (mc_rtc is tested with GCC >= 4.8.0, clang >= 3.5.0 and Visual Studio >= 2019). Your best bet is to try it out.

Thanks, @gergondet
Were there any known instances of mc_rtc getting successfully ported to OSs like QNX or VxWorks?
Thanks.

Were there any known instances of mc_rtc getting successfully ported to OSs like QNX or VxWorks?

Not that I'm aware of. From what I can gather, both VxWorks and QNX recent releases have modern toolchains and it seems Boost also supports both platforms so it should be mostly fine but it might also require some work.

Hi @magnusmusngi
I will be closing this issue. Please open a new one if you encounter build issues on QNX and/or VxWorks and we will do our best to help you.


Update embedded-svc

Update the dependencies and bump the version

In the edge-net stack and across the board in my projects, this embedded-svc is the only one on the "old" release of embedded-svc

Does it build properly with the embedded-svc feature enabled?

I've tested it with embedded-svc and ws features enabled (it builds)

I did raise edge-ws 👍

Ah yes - thanks!


Make SECURITY.md consistent

We're missing these across a lot of Knative repos, this is copied/improved from the 3 that existed.

Argh, the gh CLI is not helping me.


add prod list user

ref #632

Create zfs dataset for the k3s kubevirt storage
zfs create new_data/apps/k3s_storage

The vm boots, but doesn't get an IP and using virtctl console testvm does not result in anything. Maybe try using VNC? Or try enable more debug logs for kubevirt.

Good article for debugging https://groups.google.com/g/kubevirt-dev/c/Ezd3ZtxcxLI. Maybe the qcow is for uefi?

Find the Skyconnect USB dongle running usb-devices inside the hypervisor.

Pin nixpkgs version in nixos by using a flake https://gist.github.com/m1cr0man/8cae16037d6e779befa898bfefd36627

export VERSION=v1.1.1
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-operator.yaml
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${VERSION}/kubevirt-cr.yaml
kubectl get all -n kubevirt # poll until ready
kubectl apply -f kubevirt_cr.yml

Using this script from https://www.reddit.com/r/homeassistant/comments/11pkmpu/obtain_apple_thread_network_credentials/ to create TLV to make the Apple border router the default network.
import python_otbr_api
from python_otbr_api import PENDING_DATASET_DELAY_TIMER, tlv_parser
from python_otbr_api.pskc import compute_pskc
from python_otbr_api.tlv_parser import MeshcopTLVType, MeshcopTLVItem

# Apple
CHANNEL = <Channel Number>
PANID = "<YourPanID>"
EXTPANID = "<YourExtPanID>"
NETWORK_KEY = "<Your Network Key>"
TIMESTAMP = b'\x00\x00\x00\x00\x00\x03\x00\x00'

channel = MeshcopTLVItem(tag=0, data=CHANNEL.to_bytes(length=3, byteorder='big'))
pan_id= MeshcopTLVItem(tag=1, data=bytes.fromhex(PANID))
ext_pan_id = MeshcopTLVItem(tag=2, data=bytes.fromhex(EXTPANID))
network_key = MeshcopTLVItem(tag=5, data=bytes.fromhex(NETWORK_KEY))
timestamp = MeshcopTLVItem(tag=14, data=TIMESTAMP)

tlv_new = {0: channel, 1: pan_id, 2:ext_pan_id, 4: network_key, 14: timestamp}
tlv  = tlv_parser.encode_tlv(tlv_new)
print(tlv)

Afterwards merge the Apple and Skyconnect border routers and wait for 5 minutes.

Updated Unifi settings as follows to support IPv6 on the LAN:

Setup static ips for the Apple TVs

Waiting an entire night has recovered the home hub and associated sensors.

Ideas to improve thread performance:

disable mDNS esphome devices
Disable home hub slaapkamer tv
check if it’s due to Apple TV or sky connect

Following this article https://www.derekseaman.com/2023/10/part-3-smart-home-matter-and-thread-deep-dive.html disabled multicast in the lan and disabled multicast enhancements in WiFi

Enabled IGMP snooping

Maybe try using a separate pci / usb network dongle for haos?


Remove store

This removes the internal store type, in favor of passing around a *gorm.DB instance. This is better, because it allows us to define better transactional boundaries, so we can wrap methods on the Empire struct in a transaction. This fixes that bug (thought we had an issue for it but couldn't find it) where deploying a docker image that didn't exist would still create an app within Empire, as well as some race conditions during release creation.

:+1:


use properties for basic sketch info values, use copy instead of …

…copy constructor syntax, remove a few unnecessary ctors

Different PR to go into apache main so closing this


One blank line below a list is removed on save

Describe the bug
When I have one or more blank lines below a list (ordered or unordered list), one blank line under each list is removed each time the document is saved.  If I edit and save the document again, the next blank line under each list is removed.
To Reproduce
Steps to reproduce the behaviour:

Add a paragraph of text.
Insert three blank lines.
Add an unordered list with one or more items.
Insert three blank lines.
Add a paragraph of text.
Insert three blank lines.
Add an ordered list with one or more items.
Insert three blank lines.
Add a paragraph of text.
Save (you will notice one blank line below each list is removed)
Save (you will notice one blank line below each list is removed)

Expected behaviour
I expect the blank lines that I have added throughout the document remain in the document when saved.
Desktop (please complete the following information):

OS: Windows 10
Browser: Chrome
Version 94.0.4606.81 (Official Build) (64-bit)

Additional context
The version of the editor we have tested this on is 3.1.0.

@JasonBullockMo
Can you upload the any gif image or video for issue?

Hi @js87zz , can you please review my request?

@JasonBullockMo
Sorry for late replyung.
It looks similar to #1896 .

We have a simple prototype for testing. We use Javascript to save the markdown into a text area, which we can then copy into the clipboard.
$('.save-button').click(function () {
$("#mytextarea").text(editor.getMarkdown());
       $("#mytextarea").select();
       navigator.clipboard.writeText($("#mytextarea").val());
});

The text view mode means toastui-editor-viewer or any other things?
Here’s what we’re using as a viewer. Our Viewer prototype exists on a different page to our Editor prototype.
const editor = new toastui.Editor({
       el: document.querySelector('#editor-0'),
       initialEditType: "wysiwyg",
       hideModeSwitch: true,
       viewer: "true"
});

// We later load the Markdown from a textarea:
editor.setMarkdown($('#mytextarea').val());

@JasonBullockMo
Thanks. I checked the issue clearly.
It will be fixed.

@JasonBullockMo Thanks. I checked the issue clearly. It will be fixed.

Hi js87zz, can you please provide an update on when this is scheduled in to be fixed?


gopass show doesn't display password, even with -C

Summary
Latest release doesn't show the password anymore, even when called with -C.

OS: Arch
OS version: Latest
gopass 1.9.1

Sorry, I can't reproduce this.
For me it's working with either autoclip true or false.
Do you have more information to reproduce?

I can reproduce it and actually came here because of the same problem.
$ pass twitter.com/MetalMatze
✔ Copied twitter.com/MetalMatze to clipboard. Will clear in 10 seconds.
$ pass show twitter.com/MetalMatze
✔ Copied twitter.com/MetalMatze to clipboard. Will clear in 10 seconds.
$ pass show -o twitter.com/MetalMatze
✔ Copied twitter.com/MetalMatze to clipboard. Will clear in 10 seconds.
XXX

As you can see only show -o actually outputs the contents of the password.
This is also a breaking change in the API, which I noticed because I started piping Copied twitter.com/MetalMatze to clipboard. Will clear in 10 seconds. into my password fields:
gopass ls --flat | dmenu | xargs --no-run-if-empty gopass show | head -n 1 | xdotool type --clearmodifiers --file -

After updating on Arch to
gopass 1.9.1 (2020-05-10 11:42:40) go1.14.2 linux amd64
My autoclip settings was suddenly set to true. After changing it, it works as expected and does not print the password when issuing gopass my/secret

Same here, -c stopped working with latest update I got
1.9.0-2

@chama-chomo you do not have the latest version. The latest is 1.9.1, where the issues you are describing have been solved

After updating on Arch to
gopass 1.9.1 (2020-05-10 11:42:40) go1.14.2 linux amd64
My autoclip settings was suddenly set to true. After changing it, it works as expected and does print the password when issuing gopass my/secret
This was My case as well the autoclip was set true in *$XDG_CONFIG"  config.yml

Had exactly the same problem, flipped autoclip to false and passwords showed up again.
In the meantime it broke my scripts too, not a big deal, but I was quite perplexed at first xD

Thank you all for reporting on this issue.
In https://github.com/gopasspw/gopass/issues/1323#issuecomment-623656339 I have listed the intended behaviour and verified that HEAD still works that way. So the current implementation is working as intended.
However there seem to be two issues here:

the default for autoclip used to be true and was only recently changed to false. Whoever started using between 1.4 and 1.8 likely has autoclip set to true. In #1327 the previously inconsistent behaviour of autoclip was fixed leading to surprising behaviour for some users. This was definitely not intended, I'm afraid. However now I don't see any clean way to go back.
autoclip interacts poorly with non-interactive usage. This can probably be improved. This should be tracked in #1352.

For now please make sure to use gopass show -f in your scripts.


createRetrieverTool for chatting with my docs

For the previous version of langchain I was able to use createRetrieverTool from langchain/agents/toolkits, however now that I updated langchain to 0.1.0 it shows that its deprecated. But on the documentation it still shows to use createRetrieverTool, so how can I build a Chat with my pdf using langchain?

I see that now there is a "createRetrieverTool" available from:  "langchain/tools/retriever"

@corintho yeah I just tried that and it worked, regardless I already implemented another way of chatting with your docments without the retriever tool

const model = new ChatOpenAI({ modelName: "gpt-4", temperature: 0 });
const loader = new PDFLoader("path/to/file.pdf");
const docs = await loader.load();
const splitter = new RecursiveCharacterTextSplitter({
chunkSize: 1000,
chunkOverlap: 0,
});
const texts = await splitter.splitDocuments(docs);

const vectorStore = await FaissStore.fromDocuments(
  texts,
  new OpenAIEmbeddings()
);

const retriever = vectorStore.asRetriever();

const SYSTEM_TEMPLATE = `Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
----------------
{context}`;

const messages = [
  SystemMessagePromptTemplate.fromTemplate(SYSTEM_TEMPLATE),
  HumanMessagePromptTemplate.fromTemplate("{question}"),
];

const prompt = ChatPromptTemplate.fromMessages(messages);

const chain = RunnableSequence.from([
  {
    context: retriever.pipe(formatDocumentsAsString),
    question: new RunnablePassthrough(),
  },
  prompt,
  model,
  new StringOutputParser(),
]);

const result = await chain.invoke(msg);


REPO HAS BEEN ABANDONED - PLEASE ARCHIVE TO SIGNAL NO INTENTION TO MAINTAIN

There are a number of pending PRs and it doesn't look like anything's been edited/merged for 2 years. Has this repo been abandoned? If so, please make that clear so folks don't waste their time trying to submit improvements that will never be accepted.

Ping?

It seems like this has been abandoned.

Ping? Please abandon if you have no intention of accepting PRs or addressing pending issues.


Use executeBlocking to call the next service

As discussed in #1, here is the second PR.

Well, this code seems much more legible than the version on #1 . I propose that we merge this version instead  #1. Any objections @yanaga and @christian-posta ?

Yah agreed @rafabene
+1 to this PR

If that's the Vert.x way, then I presume it's the most readable and understandable option within the constraints. +1

This way is definitely closer to the vert.x way and it perfectly fine for vert.x. The "pure" vert.x way would be to use a non-blocking HTTP client, but unfortunately, Feign does not support this. It has an async version, but the callback is called in their own thread pool (and not the event loop) and it requires to switch to RX (which in this case it not useful, and confusing).
You may wonder why being called in the event loop is important. This is because Vert.x components are (generally, as there is rare exceptions) called by a single thread, so you don't need synchronization. As we never blocks this thread, it can handle a huge load. Not being executed by the right thread, breaks this, so you would need synchronization...

Thanks for the explanation @cescoffier . So I just merged this version and closed #1 . Thanks for the comments and votes @christian-posta @yanaga

:+1:


Code_for_hacktoberfest!

Make a pull request of whatever apps, games, html pages, code you have written for hacktoberfest.

hey afred khan, I want to make a PR of my code of my little game i created in python!
Can you help me out?!

hey... I want to make a PR of magic number code in python. So if u can help me!

Hey,
I want to make a PR on Depth First Search and Depth for Search for Graphs in python because there's room for that in your -HACKTOBERFEST2K20/Python/

I  want to make a simple portfolio website for myself. can I include that?

@Jayanth19524  Sure, you can

HEY...!!!!! i just want to make PR of Snake Water Gun game .  Can You please see and if you can add. :)

Hey! @afred-khan  I want to make a PR of DS algo questions in python. So if you can help me!

I want to make a PR of my covid application. So if you can help me!

I want to make a PR for Sudoku solver. Can I do that?

Hi, I want to make a PR for a website.Can I do it?

@VishalAmbavade @sumasreeeduru You can make a PR on any topic, go ahead!

i want to upload some linkedlist questions can i do that?

Hey, can I make a pr of Searching Algos(Binary ,Linear) in python??

Hey ! I wanted to add basic blog using django python

Hi can I make PR on Twitter Bot which I have done?

I have made a pac man game please checkout

Wow

Tried Hangman simple code

I have created music app using javascript can I add it here?

I want to add some cool games based on JavaScript
can you please assign me..?

Hello @afred-khan
I have seen that there is no program for insertion sort in the C folder
I would like to add this program in this folder
so can you assign this to me

Hello @afred-khan
I would like to add programs of link list and it's operation in C folder
Can you please assign this task to me??

@aritrochakraborty29 @Anupam-Panwar No need to get assigned for PRs, just add them.

can I add my app I made for covid19 statistics.

Where can I add my PHP Script ?

I want to add a Database Application in C, I checked that you do not have anything much other than Algorithms in C

Can I organize the code by language, because a lot of code belonging to the same language is in different folders and I feel that it should be in the following structure:
README.md
...
JavaScript
      |
      ----- folder1
      ----- folder 2
Python
     |
     ----- folder 3
     ----- folder4

In this each folder there is a different project so there wouldn't be a mixup of code and other problems.

I have done this and made a pull request

Hey, I want to make PR my Automated Folder Cleaner program.


Add KnownProxies configuration

Fixes #733.

Squash and merge.

Failure is unrelated.

Thanks for the PR!!! I wonder if our ordering for forwarded headers is in the correct spot too https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-3.1#forwarded-headers-middleware-order Also from the discussion here I wonder if we also need to have KnownNetworks (https://github.com/dotnet/AspNetCore.Docs/issues/2384) Maybe not? All the docs only mention knownproxies (https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-3.1)  I'm curious what was the scenarios/setup you tested this with (I think you mentioned docker-compose earlier)? Also this is interesting: https://github.com/dotnet/AspNetCore.Docs/issues/12349 and https://soapfault.com/2020/02/24/asp-net-core-reverse-proxy-and-x-forwarded-headers/ cc @ejsmith

I think the location is fine, if it even matters. I've created a reverse proxy in IIS and the KnownProxies must be set, otherwise .NET Core doesn't know which IP is the proxy.
KnownNetworks is only required to limit where the forwarded headers can come from.

Thank you very much for helping us with this issue!
I noticed on this blog post they are saying to clear the KnownProxies list. I wonder if that would be a better way for Exceptionless to handle this issue. I think it can be somewhat of a security concern if you are using those things for any sort of access rules. In Exceptionless I believe we are simply using them to populate information about the events. So seems like it would be OK and it would make it work in all scenarios without configuration. Thoughts?

Clearing them doesn't work with IIS, it'll need to be specified explicitly. It also says that ("To forward the scheme from the proxy in non-IIS scenarios"). Normally only the IPv6 loopback is added. Basically with this change it stays as-is but enables Exceptionless to be reverse proxied through IIS 😊

Here is a docs page about it too: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-3.1#forward-the-scheme-for-linux-and-non-iis-reverse-proxies

Clearing them doesn't work with IIS, it'll need to be specified explicitly. It also says that ("To forward the scheme from the proxy in non-IIS scenarios"). Normally only the IPv6 loopback is added. Basically with this change it stays as-is but enables Exceptionless to be reverse proxied through IIS 😊

Hmm... they seemed to indicate that it would work with IIS automatically.

I've tried that already, but it doesn't work for me unless KnownProxies is correctly specified.

Did you try adding UseIISIntegration?

Actually, that should be on already as ConfigureWebHostDefaults appears to do it.

Yeah, I've also tried that, but it doesn't help because it's not running through IIS but inside docker. If I run Exceptionless straight through IIS (i.e. hosted) it's working correctly. The block of code from https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/proxy-load-balancer?view=aspnetcore-3.1#forward-the-scheme-for-linux-and-non-iis-reverse-proxies doesn't work, I've tried it (and variations of it) several times it it's always ::ffff:<IP_ADDRESS>. This host needs to be added to KnownProxies, otherwise the wrong IP gets reported. This is exactly where KnownProxies is for.
The request's original remote IP must match an entry in the KnownProxies or KnownNetworks lists before forwarded headers are processed. This limits header spoofing by not accepting forwarders from untrusted proxies.

Ok, hmm... so my understanding is that the KnownProxies list is basically white listing entries, but if the list is empty it should work always if those headers are present. I'm just trying to understand this because ideally I don't want to have to add an extra configuration step.
Looking at the code here: https://github.com/dotnet/aspnetcore/blob/master/src/Middleware/HttpOverrides/src/ForwardedHeadersMiddleware.cs#L223
It seems like from reading that code if both of those collections are empty then it bypasses any known hosts checks and if the headers are present it should use them. But you are saying that you set both collections to empty and it did not apply your incoming X-Forwarded headers, correct?

No, this must be set for forwarding headers to be accepted. It's only working for loopback. See above.
For what it's worth, it's also possible to use KnownNetworks, e.g. options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("<IP_ADDRESS>"), 16)); But this format isn't very friendly as it's not CIDR notation. I could add this also and make it something like KnownNetworks: strings in format IP-PrefixLength

Update: clearing all Known... properties seems to work as well, let me try some more.
For what it's worth, it's also possible to use KnownNetworks, e.g. options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("<IP_ADDRESS>"), 16)); But this format isn't very friendly as it's not CIDR notation. I could add this also and make it something like KnownNetworks: strings in format IP-PrefixLength

:-) I was going to say... I read that code and I don't see how it would be checking if both are empty. I feel like in general they need to be careful with setting these things because people might be making security decisions based on their values. But in our case we are not and in general if we are able to just use these always and not have to have a config option for this, that is what I would prefer. Can you do some testing and let me know what you find out? Thank you!!

That indeed seems to work as well. I'll update to the code to clear them all by default, that seems to be enough 😊

Awesome! Thank you @PhyxionNL !!

https://github.com/exceptionless/Exceptionless/pull/736

Thanks!


RenderDoc crash when trying to capture nvidia perf counter from perf counter view

Description
RenderDoc crash when trying to capture nvidia perf counter from perf counter view
first I enabled all nvidia counters, it crashed.
then I tried to remove some nvidia counters, it still crashed
then I removed all nvidia perf counters, it worked.
Steps to reproduce
RDC capture is uploaded to here : https://renderdoc.org/bugreporter/report/Q63OEozU4l366

use Renderdoc v1.28 to open the capture
goto perf counter view,
enabled all nvdia perf counter
click sample counters
Renderdoc crashed

Environment

RenderDoc version: v1.28
Operating System: Windows10
Graphics API: DirectX11
GPU RTX NV3070
NV Driver <IP_ADDRESS>23

If this crash only happens when enabling nvidia's perf counters you will need to contact nvidia to ask for help with the problem. The crash is happening in their driver and both the integration/implementation of those counters and the library behind them are nvidia's responsibility.

thanks for the advice!


Implement num_traits::Zero, One, and Pow for Modint

I added num-traits as an optional dependency so that we may use libraries requiring Zero, One, and so on, only when we know the library is available in that environment.

Do I need to add a feature flag to the CI script as well?  I'm not sure what is the best practice to do so.

Do I need to add a feature flag to the CI script as well? I'm not sure what is the best practice to do so.

We need to do, I think.

@qryxip I tried removing .cargo/config.toml and editing all-features = true.  Am I doing correctly?
Also, I am not sure how to add a feature flag to the CI script, could you help me out?  Do I need to add a new matrix for "all features", or maybe another way?


[PT Run] Allow new settings for results tuning

Description of the new feature / enhancement
Since not all people will be happy with how the current results are displayed, we should add some "global" settings for this. There does seem to be some suggestion about per-plugin priority/score and the like, but this is not that.
Scenario when this would be used?
The current model with is great for a lot of reasons. Primarily it's fast and predictable. It's predictable in the sense that once it's selected an item from the results, async/delayed results added to the list do not alter this and so whenever you see something selected and you hit enter, that's what you get. This is a major issue with the built-in Windows search.
However, for users that have the majority of the results they want access to come from the async/delayed search results, the above falls short of being useful.
The idea is to all setting that can change this behavior in two ways to start with:

Allow changing the weight of previously selected/clicked results so that they can rise faster. This is needed because of the non-unified score model across plugins.
Allow the async/delayed results to be selected by default, so that the top/and more relevant results become the selected action when hitting enter.

Of these two, only the second is a behavior change, and it can be toggled on/off globally.
Supporting information
No response

This might also help with #17879, after some more work. However, I think the solution to solution to #17879 would be having additional weights for each plugin, so you can influence their sorting.

@jefflord
Can we please sync this with #1787 and #17879! And I am not sure if we should allow the user to configure one value for all plugins.
And maybe we should remove the special scoring setting in system plugin to not end up in multiple conflicting settings.

@jefflord Can we please sync this with #17878 and #17879! And I am not sure if we should allow the user to configure one value for all plugins.
And maybe we should remove the special scoring setting in system plugin to not end up in multiple conflicting settings.

I saw #17878 and #17879 think they are on the right path (see 3c below) but...

I don't really know what you mean by "sync" here as I am new to this.
I don't know what you mean by "allow the user to configure one value for all plugins", where is that?
I agree that #17878 is a good idea because there are three things to consider on how a result rises to the top.
a. "Score" -- though this really can only be compared to scores from the same plugin
b. "Usage history" -- have you clicked this item before
c. "Favored plugins" -- what #17878 is after.
I don't really know what is meant by the "special scoring setting in system plugin"

@jefflord Can we please sync this with #17878 and #17879! And I am not sure if we should allow the user to configure one value for all plugins.
And maybe we should remove the special scoring setting in system plugin to not end up in multiple conflicting settings.

@htcfreek
I saw #17878 and #17879 think they are on the right path (see 3c below) but...

I don't really know what you mean by "sync" here as I am new to this.

I think this can be combined with your changes as it goes in the same direction.

I don't know what you mean by "allow the user to configure one value for all plugins", where is that?
I agree that #17878 is a good idea because there are three things to consider on how a result rises to the top.
a. "Score" -- though this really can only be compared to scores from the same plugin
b. "Usage history" -- have you clicked this item before
c. "Favored plugins" -- what #17878 is after.

Your PR introduce one value that is used for all plugins. But different plugins generate the score in different ways and it might be good to allow the users to have individual score per plugin. Otherwise you only make the score higher but not change the result order.

I don't really know what is meant by the "special scoring setting in system plugin"

For system plugin we have a setting to reduce score for the ip and mac results. maybe we can remove that after improve the scoring system.
The setting was required because system plugin calculate the score based on FuzzySearch* and other plugins doesn't. So the results were scored to high compared to others. (Personal I think that this is the main reason for all our problems. If all plugins calculate the score in the same way we will hopefully get better results order.)
*) There is a bug in FuzzySearch class that we get 0 score in some cases.

@htcfreek
On the sync, OK, except I think we'd want to have  #17878 and #17879 implemented as value in the settings screen for each plugin, for the UI. Since " #17878 and #17879" do not have a UI component, I think I would sync them in "in concept" to this, and perhaps that's really what you mean. I like that, it is the same direction.
"Your PR introduce one value that is used for all plugins" yes and no. The value applies to all plugins because all can have results that can be clicked. I don't think you want a "clicked weight" to be different weight, but you certainly could combine "clicked weight" and the per-plugin score-bump from #17879. And yes, it all stems from nonuniform scores.
OK, "system plugin" is "Windows system Commands"... (I'm new here). I see score-mod option now. I think this is fine for now but yes, also agree would not be needed if we have exposed per-plugin score modifiers.
You said: "Personal I think that this is the main reason for all our problems." -- I think we'd all agree here, but that's a tough thing to tame at this point.
A larger overhaul would be needed to solve that issue, but that does not mean per-plugin weight/preference won't still be need (it will be).
For the root cause, "disparate system", I suppose we'd need a score-api. Here the plugins would not set the score, but they would return each of their results "scorable data and hints"
So Wox.Plugin.ScorableData would replace Wox.Plugin.Score. Plugins would populate ScorableData instead of raw Score. And we'd have an API that takes the ScorableData and does the sort after it has all the results from all plugins...
class ScorableData {
 string name;
 string desc;
 LocalWeight weight = LocalWeight.Normal;
...
}

Same page?

OK I cleaned up this section:

And added "Sort Weight Boost" for each/all plugins, which give you a lot more control, allowing users to push things like "Window Walker" up higher in the results.

feedback please...?

that is what i'd assume would happen.  i would still believe a weight boost would be a multiplier vs a static number but that is just me.

Honestly let’s pick one way and move forward. If it needs tweaks we can do it.

@jefflord , @crutkas
I agree with both of you. Let's keep what we have. I like your solution/implementation, @jefflord. I am interested in what our users will think about the new settings and if they will like them. And later in a second step we can unify the scoring across plugins and tweak the new scoring settings if necessary.

We released 0.60.1 and this is included in it!  https://aka.ms/installpowertoys


Improve testing structure

To start improving our test setup we want to bring the folder structure in line with common practices:
__tests__/{integration,helpers,...}
src

In other words __tests__ should be at the root not inside src. The current long running test should go under __tests__/integration as it depends on the runtime dependencies.
This opens up the possibility to create a new folder structure that mirrors the folders under src with 1 file per file in the production code for unit testing.
All tasks and build steps that depends on the current folder structure will need reviewing and adapting.
This was discussed as a first step to improve the testing with @dolcalmi.

I don't think it's wise to put __tests__ outside of src/. a lot of the config file that are within src (tsconfig, package.json, node_modules...) are necessary to run the tests. and they are within src/

All of the files you mention are in the root of the repo not within src. 🤔
Sent from my iPhone

On 30. Jun 2021, at 23:13, nicolasburtey @.***> wrote:

I don't think it's wise to put tests outside of src/. a lot of the config file that are within src (tsconfig, package.json, node_modules...) are necessary to run the tests. and they are within src/
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

To be clear I am proposing to move to this kind of structure:
$ tree repo
repo
├── __tests__
│   ├── fileTest.ts
│   ├── helpers
│   ├── integration
│   │   └── integrationTest.ts
│   └── subfolder
│       └── otherFileTest.ts
├── package.json
└── src
    ├── file.ts
    └── subfolder
        └── otherFile.ts

This is a very common structure in the node js ecosystem.

oups yeah you're right! nevermind my comment above.

Should probably use tests as the dir name instead of __tests__.

To apply the common pattern test is probably the right name.

You're right, I misremembered the name.

__tests__ is the default folder for jest AFAIK. this can be overridden of course.

I don't really care what the folder name is __tests__ or test or whatever... my point is regarding the structure.

yup ok. this would also simplify what needs to be in the production docker image. right now we have an ugly script to delete the tests folders: https://github.com/GaloyMoney/galoy/blob/9f4c4ebdcb186664f500ea5420218e3b2b38d047/Dockerfile#L13

yup ok. this would also simplify what needs to be in the production docker image. right now we have an ugly script to delete the tests folders:

Yup that's a big reason why I prefer this structure


Accents

Hi Michail,
Any idea how to get rid of the accents in capital letters?

Thanks!

In contrast to other languages like French - in Greek accents need to be dropped when the words are in uppercase. Unfortunately this needs to be done in a language-by-language basis something which  aor-language-greek can't provide since admin-on-rest  doesn't take into account this edge-case.  Your best bet is to write some javascript/react code that selects by selector "text-transform: uppercase" and make replacements on each of the letters that have accents on them.  Something like that is done from:  https://github.com/ebababi/jquery-remove-upcase-accents but could be easily implemented without jquery and an extra library

Thanks!
Your suggestions are great. May I suggest this change to the code? (so that you do not need the 'a bit dangerous' part)
const spanList = [...document.querySelectorAll("span")]
.forEach(function(el) {
etc....

Another (semi)solution for not "penalizing" performance is to have 2 translations, one of them using upper case letters for whatever AOR converts to uppercase (for example: "ΔΗΜΙΟΥΡΓΙΑ". So the user can choose which version they want.
// FILE: ourMessages.js
const ourMessages = {
    aor: {
        action: {
            delete: 'ΔΙΑΓΡΑΦΗ',
            show: 'ΕΜΦΑΝΙΣΗ',
            list: 'ΛΙΣΤΑ',
            save: 'ΑΠΟΘΗΚΕΥΣΗ',
            create: 'ΔΗΜΙΟΥΡΓΙΑ',
            edit: 'ΕΠΕΞΕΡΓΑΣΙΑ',
            cancel: 'ΑΚΥΡΩΣΗ',
            refresh: 'ΑΝΑΝΕΩΣΗ',
            add_filter: 'ΠΡΟΣΘΗΚΗ ΦΙΛΤΡΟΥ',
            remove_filter: 'ΔΙΑΓΡΑΦΗ ΦΙΛΤΡΟΥ',
            back: 'ΕΠΙΣΤΡΟΦΗ'
        },
    },
    post: {
        action:
            {
                save_and_show: 'ΑΠΟΘΗΚΕΥΣΗ',
                save_and_add: 'ΑΠΟΘΗΚΕΥΣΗ ΚΑΙ ΝΕΑ ΔΗΜΙΟΥΡΓΙΑ',
            }
    }
};

export default ourMessages;

Then, at App.js:
import greekMessages from 'aor-language-greek';
import ourMessages from './ourMessages.js';

const messages = {
    el: {...greekMessages, ...ourMessages},
};

@afilp I decided to drop the accents from the actions so that they are compatible with upper casing!
Thanks for the tip const spanList = [...document.querySelectorAll("span")] I didn't know I could do that :)

@zifnab87 Hi Michail, my linkedin is here:
https://www.linkedin.com/in/arisfilokostas/
I have made an invitation to you which is in pending status. I know you have hundreds of them (I have the same problem, many spammy invitations!), just visit the above link and you will find it. Thanks!


Update LaunchAgents extension

Description
Add list for system and loaded launch agents
Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

will u have time to review this today @pernielsentikaer ?
also, on future update, was considering updating launch agents to background tasks. do you think it would improve understanding what this is or not so much?

I think launch agents are well known as they are, don't you 🙂


How to write filtered queries

I am trying to filter a query for date range and related field values.  I assume that the syntax for BungieSearch filtered queries syntax would be derived from elasticsearch-dsl-py.  However from reading the elasticsearch-dsl-py docs I am still not able to figure out how to construct filtered queries. Any advice or nudge in toward the right documentation is welcome. Thanks in advance.

The elasticsearch-dsl-py package handles it itself actually. Just call .filter on a search object (like any Bungiesearch instance), and you'll have a filtered query.
For example:
In [2]: RawArticle.objects.search.to_dict()
Out[2]: {'query': {'match_all': {}}}
In [3]: RawArticle.objects.search.filter('term', name='TEST').to_dict()
Out[3]:
{'query': {'filtered': {'filter': {'term': {'name': 'TEST'}},
   'query': {'match_all': {}}}}}

Thanks again for the reply. Still missing something though. I'm trying to either chain the search objects filter method with the query method or somehow run a query with the generated filtered query like this
s = Song.objects.search
s.filter('term', title='Test')
results = s.query('match', _all='foo')
My goal is to search through the lyrics of songs with the query but limit the searchable dataset with filters such as term filters for the 'title' field and range filters for the 'release_date' field.
I cannot figure out how to
A. execute filtered queries after they have been generated by the 'filter' method
B. combine one or more filters with a search query to generate a result set in Django view to be then serialized and returned
Do you have examples of either of these two things, that you could paste in?
Thanks again for all your help.

To execute the query, the easiest way is to simply take a slice out of the
bungie search object using brackets, eg [2:10] will tell elastic search to
start from position 2 and retrieve a size of eight. The returned results
will be automatically mapped to the Django objects, so it actually executes
an SQL query to fetch the latest info from the database. If you want the
raw Result objects from elastic search DSL, use a slice whose third item is
True (it shouldn't be Python valid, but bungie search overwrites the
slicing function so it works), eg [2:10:True] .
Sorry for the partial example only, I'm answering from my phone.
Best regards,
Christopher Rabotin.
On Jul 21, 2015 10:55 PM, "meninoebom"<EMAIL_ADDRESS>wrote:

Thanks again for the reply. Still missing something though. I'm trying to
either chain the search objects filter method with the query method or
somehow run a query with the generated filtered query like this
s = Song.objects.search
s.filter('term', title='Test')
results = s.query('match', _all='foo')
My goal is to search through the lyrics of songs with the query but limit
the searchable dataset with filters such as term filters for the 'title'
field and range filters for the 'release_date' field.
I cannot figure out how to
A. execute filtered queries after they have been generated by the 'filter'
method
B. combine one or more filters with a search query to generate a result
set in Django view to be then serialized and returned
Do you have examples of either of these two things, that you could paste
in?
Thanks again for all your help.
—
Reply to this email directly or view it on GitHub
https://github.com/Sparrho/bungiesearch/issues/92#issuecomment-123488690
.

Thank you again for responding. I am now able to create filtered queries.
The reason I struggled with this is because in the ES dsl a 'term' must take an array:
http://stackoverflow.com/questions/30363647/elasticsearch-the-terms-filter-raise-filter-does-not-support-mediatest
For instance this query body raised an exception:
[{"query": {"filtered": {"filter": {"terms": {"title": <search_term>}}, "query": {"match": {"title": <search_term>}}}}}]
This should be added to the documentation of both BungieSearch and elasticsearch-dsl-py
I will try to send a pull request as soon as I can.

What are you trying achieve with [{"query": {"filtered": {"filter": {"terms": {"title": }}, "query": {"match": {"title": }}}}}]? It initially doesn't look like a valid JSON object for searching in my eyes. It looks like two distinct queries to me, which would require the multi query API (or something like that) which isn't supported by elasticsearch-dsl-py.
Do you want to filter on the title and query the title? If so, this may lead to odd results if the title is a string, depending on the analyzer enabled on elasticsearch. In fact, if there is any analyzer, the filter part won't filter anything, and query will do some kind of fuzzy matching (depending again on the analyzer used). If there isn't any analyzer, I expect this to return only if the title you search for matches strictly, in which case the "query": {"match": {"title": }} doesn't add anything to your request (just more resource usage on the cluster). I may be wrong in my analysis here, so check out the difference between queries and filters.
With the solution you wrote, Song.objects.search.filter("terms", title=title).query("match", title=q), you'll filter the title field on title and query that same field on q, which leads to what I wrote right above. In itself it's a valid query, but I think you may want to use a term filter instead of terms filter: the former will match exactly the variable you pass, whereas the latter will match on the list of items you provide. For example, in Song.objects.search.filter("terms", title=title1), if title1="some title", the executed query with a terms filter is ['s', 'o', 'm', 'e', ' ', 't', 'i', 't', 'l', 'e']; if using a term filter, it'll filter exactly on "some title".

My goal is to build out an endpoint for searching for terms in the lyrics field of a model called Song.
Filtering and querying the same field was a mistake.  I meant to query the lyrics field and filter by the title field. I also see that the  terms filter requires an array but the term filter does not. So now this endpoint is working:
<IP_ADDRESS>:8000/api/lyrics?q=girlfriend&title=love
Related but separate questions
Forgive me for throwing this all in a single comment, but I need help and don't want to pollute your repo with issues.
How to query for phrases
My query is simply this:
songs = s.query("match", lyrics=str(q))
Single word searches work.
But phrase searches like this...
<IP_ADDRESS>:8000/api/lyrics?q=I%20love%20you
return too many results.
I think this means lyrics field was indexed as a full-text analyzed field so that when I do a search for 'I love you' the search term is passed through and analyzer to produce a list of terms ['I', 'love', 'you']
Is there another way to write this query so that I can search for phrases?
Or should I be researching how to map the lyrics field in the index as a string?

For example:
.../api/lyrics?q=girlfriend&title=love%20me
generates the term filter
{'filter': {'term': {'title': 'love me'}}
but does not returns an empty result ([]). No even
"title": "You Must Love Me"
I assume the problem again is that the title field was mapped as an analyzed field. Is that correct? Or, is it that my filter was constructed incorrectly?
How to filter on related fields
I don't see any examples of creating filters on related fields in the BungieSearch or the elasticsearch-dsl-py docs.  This I assume is just a matter of syntax, but perhaps it is also a matter of mapping.
Include the count in the returned JSON
How can I add a field to the  JSON that gets serialized that will contain the count from the query results?
As always, thank you so much for taking the time to read and reply to these comments. Even just pointing me in the right direction in terms of docs to read would be a huge help. There are not a lot of resources for utilizing the wrappers written for ES.

Indeed, the situation you're facing is actually one we've had at Sparrho.
You need to reindex at least one field which can be filtered, and at worse,
the whole database of articles. Depending on your dataset and ES cluster,
it may take a few days.
Effectively, a filter will exactly filter (think of coffee filter) the
words you type, without any kind of wildcard. It's great for filtering by
ID for example. If you're filtering by a string, you have to make sure it
exactly is what you're looking for. So in the case of "I love you", it'll
only filter on titles that exactly match that, no trailing character and no
leading character. If you need a leading or trailing character, you should
check the prefix or regexp filters. They should do the trick.
From the whole issue you wrote, I reckon the best strategy is to add
another field to the ModelIndex class. In fact, add a StringField which
has, as parameters, 'analyzer': None (I think, double check with the Readme
file), and then reindex your dataset. Then you'll be able to filter exactly
the keywords you provide.
I hope this makes sense. I'll double check in with the morning (UK time,
where I'm currently based).
Best regards,
Christopher Rabotin.
On Jul 24, 2015 9:13 PM, "meninoebom"<EMAIL_ADDRESS>wrote:

My goal is to build out an endpoint for searching for terms in the lyrics
field of a model called Song.
Filtering and querying the same field was a mistake. I meant to query the
lyrics field and filter by the title field. I also see that the
terms filter requires an array but the term filter does not. So now
this endpoint is working:
<IP_ADDRESS>:8000/api/lyrics?q=girlfriend&title=love
Related but separate questions
Forgive me for throwing this all in a single comment, but I need help and
don't want to pollute your repo with issues.
How to query for phrases
My query is simply this:
songs = s.query("match", lyrics=str(q))
Single word searches work.
But phrase searches like this...
<IP_ADDRESS>:8000/api/lyrics?q=I%20love%20you
return too many results.
I think this means lyrics field was indexed as a full-text analyzed
field so that when I do a search for 'I love you' the search term is
passed through and analyzer to produce a list of terms ['I', 'love',
'you']
Is there another way to write this query so that I can search for phrases?
Or should I be researching how to map the lyrics field in the index as a
string?

For example:
.../api/lyrics?q=girlfriend&title=love%20me
generates the term filter
{'filter': {'term': {'title': 'love me'}}
but does not returns an empty result ([]). No even
"title": "You Must Love Me"
I assume the problem again is that the title field was mapped as an
analyzed field. Is that correct? Or, is it that my filter was constructed
incorrectly?
How to filter on related fields
I don't see any examples of creating filters on related fields in the
BungieSearch or the elasticsearch-dsl-py docs. This I assume is just
a matter of syntax, but perhaps it is also a matter of mapping.
Include the count in the returned JSON
How can I add a field to the JSON that gets serialized that will contain
the count from the query results?
As always, thank you so much for taking the time to read and reply to
these comments. Even just pointing me in the right direction in terms of
docs to read would be a huge help. There are not a lot of resources for
utilizing the wrappers written for ES.
—
Reply to this email directly or view it on GitHub
https://github.com/Sparrho/bungiesearch/issues/92#issuecomment-124704172
.

How to query for phrases
To query for phrases, the best is to use the match_phrase query. If I'm not mistaken, elastic search will search for all of the words as seen together. As you correctly pointed out, a match query will search for all the terms independently of each other. In the case of "I love you", the returned items will have the words independently, but possibly not next to each other, e.g. "I code. Github loves git. You were right".
How to filter for phrases
Filters only work on fields which are not analyzed: is your title field analyzed? By default, StringFields in ModelIndex subclasses (which bungiesearch uses to create the mapping on ES) have the snowball analyzer enabled. Hence, you can't apply string filters on them.
If you want to filter on string titles, you can either drop the index and re-index your content after specifying in the ModelIndex that you want the title field to be non analyzed, or you can add a field in your ModelIndex which will be mapped to the title but not be analyzed, e.g. title_filter = StringField(model_attr='title', analyzer=None). Then you have to update the mapping with the search_index command and --update-mapping flag, and then reindex your dataset. After these steps, you'll be able to apply any kind of filter on that field, e.g. .filter('term', title_filter='I love you').
Important note
Filters do exact matching. For example, term-filtering on "lover" will not return any document which has the word "love" or "lovers".
How to filter on related fields
Do you mean related field on Django, like foreign keys? The way to do this is by adding a field to your ModelIndex with an eval_as parameter that you then code up to retrieve and format what you'd like indexed from the relation, cf. the meta_data field in https://github.com/Sparrho/bungiesearch#modelindex.
Include the count in the returned JSON
Note: I'm not 100% sure I understood the question, so let me know if this actually does answer your question.
If you want the number of results found on ES for your query/filter, you can do a len(search_object) where search_object is a Bungiesearch object with your query, filter, and all that, cf. this example. Note that this will send the query to ES on the _count API endpoint, which requires ES to execute your search (or at least partially since that endpoint doesn't take into consideration stuff like min_score, nor does it fetch any documents).
Anyhow, if you want to avoid ES from having to do a count after you just executed that same query (which can be useful if you run very resource intense queries for example), you'll want to get the raw results from ES, which contains the hits, and then map the results. For example:
srch = Article.objects.search.query('match', _all='Description')
srch_rslts = srch[0:20:True] # The True will skip automatic mapping.
print 'Found {} results.'.format(srch_rslts.hits.total)
mapped = Bungiesearch.map_raw_results(srch_rslts) # Contains a list of Article objects.

I hope this helps more than my succinct answer of yesterday.

Handled in #91 by @meninoebom . Thanks =)


Create a ESM Script with attributes

Scripts components have a create() method for instantiating scripts with attributes.
scripts.create('scriptName', { attributes: { myNum: 1 })`

This depends on an attribute schema existing. ESM Scripts use JSDoc tags for their schema which is unavailable in engine only environments, meaning attributes cannot be set on a script
For ESM scripts in engine only, properties must be set after a script is instantiated:
const script = scripts.create('scriptName')`
script.myNum = 1

Ideally properties can also be set in-line on a script instance, at the same time as it's created.
Possible solutions
1. Re-use attribute property
In the editor, a schema will exist, so it will be used to create the attributes, when it doesn't, the values are simply assigned as properties onto the script instance
scripts.create('scriptName', { attributes: { myNum: 1 }})

Pros
Simple. Re-uses existing api.

Cons
Inconsistent behaviour between editor/engine. - In editor, this will validate and deep copy values. In engine, it will not.
semantics Attributes don't inherently exist in engine only, so this is overloading the term.

2. Separate method for engine only
Create a new method eg scripts.createWithProperties('scriptName', { myNum: 1 }) (TBD) that assigns values when instantiated
scripts.createWithProperties('scriptName', { myNum: 1 })

Pros
Clear and separate behaviour. Does not overload the attributes property

Cons
Creates an additional method which could lead to confusion. create() still exists and would work for instantiating a script with no props/attributes

3. Create an additional property
Attributes are applied first if they and schema exist, and properties are then assigned after.
scripts.create('scriptName', { properties: { myNum: 1 }})

Pros
No additional method. Clear distinction between behaviour of attributes and properties (attributes require schema, props do not)

Cons
Adds additional  property. ??

Any thoughts @mvaligursky @willeastcott @Maksims @slimbuck

How about options 4:

use the function you proposed in option 1 for the engine users:

scripts.create('scriptName', { attributes: { myNum: 1 }})

this way we preserve API compatibility with scripts 2.0

this could just assign the attributes to properties in the class, but ideally it would do a deep copy. It should be reasonably easy to have a generic deep copy function which handles numbers, strings, classes, objects, arrays and similar.

create a new function for use by the Editor where the schema is used, for example

scripts.createWithSchema('scriptName', ...);

this is never used by the engine users, and Editor users never call this manually, it's just called behind the scenes for them
so this function would not be exposed as a public engine API
this would use the schema in the editor

Hey that makes sense.
So how about the engine users have access to
scripts.create('scriptName', { properties: { myNum: 1 }})
and the Editor uses internal API
scripts.createWithAttributes('scriptName', { attributes: { myNum: 1 }})
when the engine users pass attributes to .create function, it should error out. This would help them port code from 2.0 to ESM.
Regarding deep copy. That would still be great for Vec3 and similar types, but not for Entities and similar .. not sure what would be the best here.

Single function is fine too I guess .. as long as when we don't have a schema, we Debug.warn or similar when the attributes are used, instead of just quietly ignoring them.
I don't like the extra parameter that handles cloning. That's equivalent of not cloning internally and just asking user to clone. They can do it on their side completely in that case. We either clone by default, or let them clone. Helpers are not needed?

Single function is fine too I guess .. as long as when we don't have a schema, we Debug.warn or similar when the attributes are used, instead of just quietly ignoring them.

Yep agreed.

I don't like the extra parameter that handles cloning. That's equivalent of not cloning internally and just asking user to clone. They can do it on their side completely in that case. We either clone by default, or let them clone. Helpers are not needed?

Yep, but this would happen before onInitialized() so you can ensure things are setup. Users might want to shallow clone, or use some else, so probably a good idea to give flexibility but default to the cheapest option?

The user can clone properties any way it wants to before passing it to .create function. Not sure what the advantage is in cloning them for them within the function using a callback they provide?

Yep, that's fair, if we don't clone internally and just make it clear that the properties are assigned

The important aspect of this is how engine-only users can easily save/load attribute data from the entity's script component.
Here is an example of ordinary script attributes on entity:
{
    "attrs": [
        {
            "name": "vertex_position",
            "semantic": "POSITION"
        },
        {
            "name": "vertex_normal",
            "semantic": "NORMAL"
        },
        {
            "name": "vertex_uv0",
            "semantic": "TEXCOORD0"
        },
        {
            "name": "vertex_bone_index",
            "semantic": "BLENDINDICES"
        }
    ],
    "vertex": 177287037,
    "fragment": 177287044,
    "textures": [
        {
            "name": "texture_albedo",
            "asset": 182217219
        }
    ],
    "depthTest": true,
    "depthWrite": true
}

And script has this attribute definition:
CustomMaterial.attributes.add('vertex', { type: 'asset', assetType: 'shader' });
CustomMaterial.attributes.add('fragment', { type: 'asset', assetType: 'shader' });

CustomMaterial.attributes.add('depthTest', { type: 'boolean', default: true });
CustomMaterial.attributes.add('depthWrite', { type: 'boolean', default: true });

CustomMaterial.attributes.add('attrs', {
    type: 'json',
    array: true,
    schema: [{
        name: 'name',
        type: 'string'
    }, {
        name: 'semantic',
        type: 'string',
        enum: [
            { 'position': pc.SEMANTIC_POSITION },
            { 'normal': pc.SEMANTIC_NORMAL },
            { 'color': pc.SEMANTIC_COLOR },
            { 'uv0': pc.SEMANTIC_TEXCOORD0 },
            { 'blend-indices': pc.SEMANTIC_BLENDINDICES },
            { 'blend-weight': pc.SEMANTIC_BLENDWEIGHT },
            { '12': pc.SEMANTIC_ATTR12 },
            { '13': pc.SEMANTIC_ATTR13 },
            { '14': pc.SEMANTIC_ATTR14 },
            { '15': pc.SEMANTIC_ATTR15 }
        ]
    }]
});

CustomMaterial.attributes.add('textures', {
    type: 'json',
    array: true,
    schema: [{
        name: 'name',
        type: 'string'
    }, {
        name: 'asset',
        type: 'asset',
        assetType: 'texture'
    }]
});

Will the user be able to simply add script to entity and provide JSON mentioned above with a new system, and expect that attributes will be in correct (by the schema) types?

I'm oversimplifying here, but ignoring events + asset/entity resolution etc, the attribute system...

Exposes class members to the editor
Validates type

In an Engine only scenario, 1 is unnecessary. For 2, we should really lean on early type validation in the editor, so that type errors can be flagged before running. One issue with scripts.create(Class, { data }) is that the editor can't validate the data against the Class types. You lose any type information, whereas using scripts.add(new Class(data)) any type errors get flagged in the editor ahead-of-time.
I'd need to check what the implications of this would be, but it feels like a valid option


This only works because of a bug in dmd, and it was probably supposed to...

... be a char literal.

Got it.

Thanks.

Right back atcha!


Add ability to hide runs in the UI

This PR fixes issue #364 . I have added a gear icon inside the Alert box instead of the run name. Further, we are now hiding the run from the menu panel. You can check the UI for this commit below.
Updated screenshot No.2:

Do let me know if this UI looks fine or if I should try something else.

@kamalesh0406 Please also update the screenshot from the description to reflect your recent changes.

@kamalesh0406 Please also update the screenshot from the description to reflect your recent changes.

Done, sorry for the delay I was caught with up something else.

@chance-an can you merge the commit, thanks!

This is amazing, thank you @kamalesh0406.
As a follow-up item, could we also hide the v0, v1 artifact nodes? They do not correspond to artifacts that users have named in their code, they are implementation details of _make_list and _make_tuple.
cc @chance-an

Yup, I think that makes sense. Should we add the same gear icon for them or use something different?
Can we add this as a sub-task to #364, there are some chances that I might not be able to get to it on time since I have school starting again.


Create guide for Nether Roof map

inside your BlueMap/maps/ folder, there should be a world_nether.conf file (or similarly named). duplicate that, and call the other one world_nether_roof.conf (or similar)
inside the normal world_nether.conf file, set render-edges to false, and set max-y to 126
this makes it possible to see through the roof, into the cavey nether inside, and it prevents rendering the stuff on the roof
inside the duplicated world_nether_roof.conf file, set the name to "Nether Roof", so you can make out the different maps on your BlueMap website
uncomment (remove the # in front of) and set the min-y to 127, and comment out the max-y (put a # in front of it
this makes sure only the nether roof is rendered, and not anything else
do /bluemap reload and then do /bluemap purge world_nether, to rerender the normal nether map with your newly changed settings

now both of your nether maps should be rendering properly
you can check their progress by running /bluemap
you can also go onto the bluemap website already, and click the Update Map button in the menu to see the rendered areas

Gotta clean this up a bit, and then create an actual file for it in the community wiki

I suggest that you seperate the guide for the Nether roof and the other variation of a Nether map. They don't belong together.
Because I was really confused at step 2, setting all the values, until I realised that it would change my existing map.
The default Bluemap Nether is fine for most users and I think it looks better than the one without render-edges.

@MightyKnight Fair point. I'll make note of that for the final version of the guide. I probably will split them apart at least a little bit.
Maybe make the modification of the original map and optional step...

Todo: double check compatibility with BlueMap 5


Update qlcplus_de_DE.ts

Changed a wrong translation from "Messungen" to "Meter"

See, that was easy ! :+1:


"Sync columns from source" broken for complex datasource names

Our virtual datasources get automatically created with complex names like Untitled Query 2 08/10/2021 0... to make them unique. The "Sync columns from source" feature seems to be broken because that name gets split up in the url datasource/external_metadata_by_name/table/Presto/null/Untitled%20Query%202%2008/10/2021%2009:15:26/.
It looks like maybe this happened when changing datasource metadata updating to pull using the name instead of the id (here). CC: @zhaoyongjie
Expected results
Datasource should update
Actual results
Error
How to reproduce the bug

Create a virtual datasource with / in the name
Try to update the datasource and "sync columns from source"

Environment
(please complete the following information):

superset version: master
python version: python --version
node.js version: node -v
any feature flags active:

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

@jinghua-qa let's update our test case with / in virtual dataset name for the Sync columns from source(?) thanks!
@zhaoyongjie let's get it fixed asap as it's a regression.

Our virtual datasources get automatically created with complex names like Untitled Query 2 08/10/2021 0... to make them unique. The "Sync columns from source" feature seems to be broken because that name gets split up in the url datasource/external_metadata_by_name/table/Presto/null/Untitled%20Query%202%2008/10/2021%2009:15:26/.
It looks like maybe this happened when changing datasource metadata updating to pull using the name instead of the id (here). CC: @zhaoyongjie
Expected results
Datasource should update
Actual results
Error
How to reproduce the bug
1. Create a virtual datasource with `/` in the name

2. Try to update the datasource and "sync columns from source"

Environment
(please complete the following information):
* superset version: `master`

* python version: `python --version`

* node.js version: `node -v`

* any feature flags active:

Checklist
Make sure to follow these steps before submitting your issue - thank you!
* [x]  I have checked the superset logs for python stacktraces and included it here as text if there are any.

* [ ]  I have reproduced the issue with at least the latest released version of superset.

* [x]  I have checked the issue tracker for the same issue and I haven't found one similar.

Thanks for fille this issue. On it.


Fix content of People of CNDA page

Currently, the images look weird and small. Also, the content is duplicated.

After removing duplication and using just markdown.
https://30db3f05.cnda-help.pages.dev/About_the_CNDA/People_of_the_CNDA.html

After adding some CSS styles.
@stl-steve-moore feel free to play with the styles if you don't like my taste. 😉


Unintentional duplicate upload

Browser: Chrome
How to reproduce:
Step 1: Drop file A and file B (2 random files/images) into drop area
Result: File A and B are uploaded
Expected: File A and B are uploaded
Step 2: Drop file A into drop area again
Result: File A and B are uploaded (Error)
Expected: Only file A is uploaded
Best regard
Behamin

Nevermind can't repoduce it anymore. Guess it was a Error 40


Setting up traccar online

Hello guys, yesterday I start with Traccar, just setup one GPS to my windows 10 computer and working charm... now I have a little issue and I don't know how or where to start... I wanna track the gps from my celular or from another computer, I try to open in another computer the static ip of my windows 10 computer but can't get in... what I have to do? How I can turn my computer with windows 10 and static ip into a server or something like that to acces traccar over internet? thanks in advance for the time

You should be able to access Traccar server remotely. If it doesn't work, most likely you have some network issue (e.g. router, firewall config).

Rigth just open port in router and works now i cant conect the ios app on startup ask for server input ip and conection failed... need another port for traccar manager ios app?

What URL are you trying?

Http://<IP_ADDRESS>:8082 works on web not working on ios app

It should work fine.

Yes now is working fine i did nothing just a few moment after i wrote this post start working the ios app
Thank you
Enviado desde mi iPhone
El 06/10/2017, a las 8:31 a.m., Anton Tananaev <notifications<EMAIL_ADDRESS>escribió:
It should work fine.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/tananaev/traccar/issues/3569#issuecomment-334755219, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFWWGe9I6tTxNvx0LJ-yHlsZ-Aruxqq9ks5spiuKgaJpZM4PvvzR.


Add message during login and unlock

A message should be displayed during login and unlock (above the password field) to indicate to the user that the login/unlock time can take up to 40 seconds.
This would help reduce complains from users who maybe have less performant computer. They would at least be aware that such a long waiting time can be expected.

Big +1 to this. Loads of things we can do here, a message is a good start.
I wonder if it'd have more user-distracting affect appearing a few seconds into the duration, rather than right away. What do you think? Perhaps a different message if it does take the extreme times of 20s+? 🤔
cc @eugeniadigon any fun ideas how we might improve the perceived speed of the unlock time? Satoshi quotes? Snake game? Stats about the blockchain? 🧠

I'd advise against adding a fixed message that always says "20+ / 40+ seconds to log in" even before the user logged in - it sounds quite discouraging, even more if it's not the general rule
I like more the idea of leaving as is, but when the wallet detects it's taking more than let's say 5 seconds, that's when we display a "We're fetching your data, login can take a while" (or something like that) and then it starts swapping that message for some random / interesting facts and stats. Some examples

It's been ### days since Satoshi's last message to the world
Bitcoin price went up % in the last 24hs
There's ### apps in the Stacks ecosystem you can interact with
Even memes or easter-eggs

I'd also advice against anything related to a game because anyway 40 seconds it's not that much of a time for the user to perform an action/play something

Wasn't entirely serious about a snake game. Agree we shouldn't mention times, we can't predict these. 40s case shouldn't ever happen, something is wrong if so. It should take no more than ~3–6s.
Encourage you to experiment with these ideas Beguene, would love to see some in action.

@kyranjamie @eugeniadigon Thanks a lot for the feedback !  Those are great ideas.
I will implement this dynamic message display you suggested @kyranjamie with the copy of @eugeniadigon. It will also be configurable. We can experiment with that.

Is this issue in development?

@timstackblock @Eshwari007 I intentionally added a delay during login and unlock so you can see the messages. It will always take at least 30 seconds

@kyranjamie @beguene I am getting a white screen when trying to use this wallet on stacks art lets make sure we regression test with normal lock unlock times.


possible read of un-owned memory in is_valid_*_atom

In the state machine on level 0, if a atom (ture,false,null) is found it copies the source string to a new buffer of at least 32 bytes of size - as far as I understand to ensure that the is_valid_*_atom functions won't try to read memory that doesn't belong to us.
This check is skipped for nested atoms, however it seems that they too could run into the situation of reading beyond their space.
Take the json [null] for example. In the state machine it would transition to root(offset 0)->array_begin(offset  1) and here hit the 'n' case that calls is_valid_null_atom(buf + 1)  (pointing to the n in [null].
Now is_valid_null_atom will do it's thing and as part of that copy 64 bit (8 byte) of data from buf + 1 that would mean null]___ (_denoting unowned memory).

@Licenser
As per the code you point out to, the memcpy is done for the following reason...
// we need to make a copy to make sure that the string is NULL terminated.

https://github.com/lemire/simdjson/blob/master/src/stage2_build_tape.cpp#L194
I am not sure what you mean by "unowned memory". The code ensures that we can read at least SIMDJSON_PADDING bytes above the advertised length of the buffer (doing a memory reallocation if needed)...
https://github.com/lemire/simdjson/blob/master/src/jsonparser.cpp#L12-L41
What you cannot do is write to this memory (anything beyond the advertised length) ... that would be unsafe... but reading it is fine (we take the necessary precaution).
It is unsafe to write to this memory because we do not know what is written there, if anything. It could be used by any other code in the same process... but we do not care about it because we do not make use of these read bytes... they are ignored.
The reason we need to make a copy in some cases, is that we want to be able to write...

Note to self: this should be documented more carefully.

Oh, I didn't see the relocation in the JSON parser :) I just stumbled over this with the rust range checking. Thanks for taking the time of explaining why it is safe!

I am re-opening this since I think it needs to get documented.
Note that the same 'trick' can work in Rust. In the worst case, you may need to memcpy the input data. But even if you need to do that... the cost of a copy + mem. alloc. is really not that bad. Of course, you increase memory usage which is bad... but computationally, it is not that bad.

👍 I went for "closed as it answered my question" it's easier to re-open then to have a ticket open forever that no one knows when to close :D

This is fixed with better comments and static_asserts in a branch about to be merged.


question about ssn

I found that in the tasks config of ssn of nuscenes, you put traffic_cones and barrier together unlike the config of centerpoint. But the size difference between them is quite far. I trained the ssn model and the ap of traffic_cones is quite low. Futhermore, i use fpn+regnet+ssn by modify your code, and found that the ap of other classes is higher than fpn+regnet, but traffic_cones and peds is  lower. So i put the traffic_cones and peds in the same task, their ap are rise but still lower than fpn+regnet. Can you tell me why this happens?

The original implementation did not play with too many combinations of groups. Most of these settings are from intuition. For example, traffic cones and barriers are more likely to locate by the road while pedestrians could walk freely in a urban scene. Of course, taking consideration of the objects average size is also a good criterion/intuition to group different categories and you can feel free to have a try.
As for the performance gap in terms of these classes, first you would better to compare it with a model with comparable performance and see which class leads to the gap. Then you can also try different settings of the head (like different shared_conv_channels and shared_conv_strides) apart from different combinations of classes to achieve a better detector.

Feel free to reopen it if there is any further question.


Expand the templating system

This PR expands the templating system by introducing a rich wrapper around the Liquid Templating Language.
The templates/ directory storing the templates in the root of the crates is structured as follow: each folder in its root represents an amethyst version, containing the templates this version and onward will use.
For example, if we have the folders "0.6.0" and "0.7.1", templates for versions 0.6.5 or 0.7.0 will be searched in the folder "0.6.0", and templates for versions 0.7.1 or 0.8.0 will be searched in the folder "0.7.1", following semver ordering.
Those version folders contain subfolders containing the different template they describe. As of now, the only template is "main" designating the template for a new amethyst project, but in the future it will also include templates such as a minimal iOS app bundle. Note that a version folder does not have to contain a copy of all template types, as if one has not changed, it can be retrieved from previous versions. Templates need to be added only when they are actually changed. If it is deemed appropriate, this behavior can be used to set a high default version without updating the templates by creating an empty folder for this version.
In order to support contextual data in template files, this PR uses the crate liquid to handle templating. Files with extension GDPU (God Damn Pink Unicorn, specifications courtesy of @Rhuagh) will be processed as a liquid template, allowing dynamic content in template files. The extension is then removed in the output. Note that the parameter amethyst_version will automatically be included to liquid parameters, containing the actually requested version. Please also note that for a text file to have its line endings be correctly chosen depending on the OS, it needs to have the GDPU extension, but it does not have to actually take advantage of parsing.
In order to deploy a template, this PR provides the function deploy, taking as parameters a template name, an optional version (if none provided, it will take the latest available), a path to the output folder, liquid parameters (type exported by template.rs for convenience) and a renaming HashMap.
The renaming HashMap acts as a dictionary between filenames in the template and filenames in the output. For example, if it contains the entry ("__Cargo__.toml", "Cargo.toml"), all files named __Cargo__.toml in the template will be renamed Cargo.toml in the output. This is useful when names perturbate the integrity of the repository, or when they depend on contextual data. Note that the translation is done before the search for GDPU extensions.
Templates support all file types and directory structures. symlinks may be supported while not tested. Example usage of all of this can be found in new.rs.

Is this ready for review or still requires work?

It's been ready for a week now ahah

bors r+


removing MANY_OR DSL construct.

because:
A. It provides inconsistent API
there is a MANY_OR but not AT_LEAST_ONE_OR
B. It is used to reduce verbosity in the DSL. (nesting only OR inside MANY)
MANY(LA_FUNC, () => {
OR([
{WHEN:LA_FUNC1 THEN_DO:XXX},
{WHEN:LA_FUNC2 THEN_DO:YYY},
{WHEN:LA_FUNC3 THEN_DO:ZZZ}
...
]
})
but once all the LA_FUNCs can be computed automatically on the fly
it will look much more "pretty"
MANY(() => {
OR([
{ALT:XXX},
{ALT:YYY},
{ALT:ZZZ}
]})
C. The third reason for MANY_OR is performance optimizations.
to avoid checking the lookahead conditions twice.
once when entering the MANY and once when choosing the
alternative in the OR.
the counter argument for this is:

make it pretty first and fast second.
it could probably still be made fast by remembering the
result of the lookahead condition without adding new DSL syntax.

{ALT1:1}

Coverage remained the same at 100.0% when pulling afb70d0c0b294fa1c9bac67939427939ea0f9425 on remove_many_or into f6a4db2436714bbdeb7a1c91a6bdf746d9276bac on master.


Unable to configure retry behavior for status codes 400-499

In BasicCrawler._should_retry_request, we always return False for status codes 400-499.
    def _should_retry_request(self, context: BasicCrawlingContext, error: Exception) -> bool:
        if context.request.no_retry:
            return False

        # Do not retry on client errors.
        if isinstance(error, HttpStatusCodeError) and is_status_code_client_error(error.status_code):
            return False

The is_status_code_client_error function enforces this behavior:
def is_status_code_client_error(value: int) -> bool:
    """Returns `True` for 4xx status codes, `False` otherwise."""
    return 400 <= value <= 499  # noqa: PLR2004

This makes it impossible to configure retry behavior for these codes.

Related to #708

Having this issue with my spider, errors 403 and 406 from the target website are impossible to retry. Is there any way to bypass this issue for now? I instantiate a RequestQueue within the failed request handler but it doesn't seem to requeue the request, even if I change its unique_key
`    @crawler.failed_request_handler
async def failed_request_handler(context: Union[ParselCrawlingContext, BasicCrawlingContext], error: Exception) -> None:
context.log.error(f"failed_request_handler is processing {str(error)} for url {context.request.url} ...")
    if isinstance(error, HttpStatusCodeError):
        if context.session:
            context.log.error("retiring session ...")
            context.session.retire()

        timestamp = datetime.now().strftime('%Y%m%d%H%M%S')
        rq = await RequestQueue.open(name="rq")
        req = Request.from_url(url=context.request.url, label=context.request.label, unique_key=f"{context.request.unique_key}_retry_{timestamp}")
        await rq.add_request(req, forefront=True)

        context.log.error("requeue...")

        failed = {
            "id": context.request.id,
            "url": context.request.url,
            "error": str(error),
            "request": context.request.model_dump()
        }
        dataset = await Dataset.open(name="errors")
        await dataset.push_data(failed)

`

Hey @monk3yd, regarding your failed_request_handler. I have a few questions about the RequestQueue:

Are you using the same named queue RequestQueue.open(name="rq") in your crawler, or do you only open it inside the handler?
Note that if you don't explicitly pass a named queue to the crawler, it uses the default queue (you can open it using just RequestQueue.open()).

Hey @Mantisus, thank you for your response! I finally patched the issue by doing this
def is_status_code_client_error(value: int) -> bool:     """ReturnsTruefor 4xx status codes,Falseotherwise."""     if value == 403 or value == 406:         value = 500     return 400 <= value <= 499  # noqa: PLR2004
and something like #708
if context.session and context.session.is_blocked_status_code(     status_code=status_code,     additional_blocked_status_codes=[403, 406], ) and status_code not in self._http_client._ignore_http_error_status_codes :
This will enable retry for this status codes, though its a pretty hacky fix hehe

Hey @monk3yd.
Yes, this approach would work too.
Hopefully, when we're done with #812, you won't have to do something like this. :sweat_smile:


all: use explicit URLs to refer to old Starlark issues

GitHub allows individual issues to be transferred but provides no bulk
mechanism---also, we've started reusing issue numbers---so I
transferred the few open issues and left the closed ones.

Thanks, I'll stick to URLs for clarity.


Feature Addition request- Add AppBar widget

Field
Description

About
🚀 Flutter's Widget Wonderland! Explore, Learn, Create. Your one-stop for all things Flutter widgets! 📱✨

Label
Add AppBar Widget

Issue Title
Add AppBar Widget
Description
Overview
This issue is about adding an "AppBar" widget to the FlutterFolio widget catalog app. The "AppBar" widget is a common UI component in Flutter for displaying the app's title, leading actions, and actions.
Adhere to the below folder structure for reference
- lib/
  - screens/
    - home_screen.dart
    - widget_detail_screen.dart
  - models/
    - widget_models.dart
  - main.dart
  - assets/
  - images/
    - flutter_logo.png
  - data/
    - widgets_data.dart
  - theme
    - app_theme.dart
- README.md

Task

Create a new widget for the "AppBar" in the lib/widgets folder.
Implement the "AppBar" widget with the ability to customize its title, leading actions, and actions.
Add the "AppBar" widget to the widget_data.dart data source with the widget's name, description, and image path.
Update the home_screen.dart to dynamically generate an elevated button for the "AppBar" widget.

Acceptance Criteria

The "AppBar" widget is created and can display a customizable title, leading actions, and actions.
It is added to the widget catalog with a proper description and an image that represents the widget.
Users can access information about the "AppBar" widget through the app's user interface.

Definition of Done

The "AppBar" widget code is clean, well-documented, and follows Flutter best practices.
The widget is tested and functions correctly.
The widget is accessible through the app's user interface.
The issue is reviewed and accepted.

Please assign this issue to me.


Shaman Enhancement

I test the Shaman Enhancement with GSE 2.0.6 and my dps its only 164k , my ilvl is 865
please help

change your macro. play with it, tweek it. macros will probably vary from system to system... latency to latency. this is probably better asked in different forum about opt your dps. we are trying to iron out bugs

Hi gvtoet
You need to talk to people who play shaman.  The included shaman macro is a sample only as a starting point and not expected to be awesome.
https:///wowlazymacro.com is your friend.


Spaceship : Why not using appstoreconnect apis for GET and POST?

Question Checklist

[Yes ] Updated fastlane to the latest version
[Yes ] I read the Contribution Guidelines
[Yes] I read docs.fastlane.tools
[Yes ] I searched for existing GitHub issues

Question Subject

Spaceship GET https://api.appstoreconnect.apple.com/v1/devices exists
but why not POST https://api.appstoreconnect.apple.com/v1/devices
Question Description

For eg to get a list of devices and add a device, since appstoreconnect apis provide
GET https://api.appstoreconnect.apple.com/v1/devices and
POST https://api.appstoreconnect.apple.com/v1/devices
am trying to understand why spaceship uses
appstoreconnect apis for : GET https://api.appstoreconnect.apple.com/v1/devices
but dev portal apis for: POST https://api.appstoreconnect.apple.com/v1/devices
why not use the one so that just providing the JWT will be enough to get all the data and perform all the updates required?
Also will not need so many URLs to be whitelisted in enterprise proxy/firewall?

Maybe we can help by contributing to move it faster on the appstoreconnect api. if that is what is the reason for not moving to the POST as described above

@inforeqd512 feel free to open a pull request and add the desired functionality.


Enable titus-seccomp-agent for sshd

titus-seccomp-agent is used to intercept syscalls
for the main process of a container.
It would be confusing if, when you ssh'd to that container
those same syscalls didn't work!
This change makes it so sshd-spawned processes behave the same.
Maybe in the future we'll have other nsentered processes
also utilize TSA.
This makes the seccomp notification slighty more robust, instead
of needing full sidecar ordering.

I have a different opinion about how we should build this, I opened a pull request here to show you.

I have a different opinion about how we should build this, I opened a pull request here to show you.

Thank you. I see you understand cmake a lot better than I do.
I'll incorporate your changes.