[QUESTION] How to get provider properties within provider instance

I've gone through all of the source for Ts.ED and am stuck trying to get properties from a provider.
Here's some sample code:
import {GlobalProviders, InjectorService, Provider} from "@tsed/di";
import {QueueType} from "../types/Queues/QueueTypes";

GlobalProviders.createRegistry("WORKER", Provider, {
	injectable: true
});

export const registerWorker = GlobalProviders.createRegisterFn("WORKER");

export function BullMQWorker(associatedQueue: QueueType): ClassDecorator {
	console.log(associatedQueue);
	return (target) => {
		registerWorker({
			provide: target,
			deps: [InjectorService],
			queue: associatedQueue
		});
	};
}

In my abstract class, I need to get the queue that was passed in my decorator.
export abstract class Worker<T> implements IWorker<T>, OnReady {
	logger: Logger = CustomLoggerSetup(new Logger(nameOfClass(this)));
	name = nameOfClass(this);
	queue = Store.from(this);
	queueService: QueueService;
	abstract run(job: Job<T>): Promise<WorkerRunResponseType>;

	@Inject()
	googleStorageService: GoogleStorageService;
	@Inject()
	injectorService: InjectorService;

	public async $onReady(): Promise<any> {
		console.log(this);
		console.log(this.queue);
		this.queueService = this.injectorService.get<QueueService>(QueueService)!;
	}
}

An example of how I create a worker is here:
@BullMQWorker(QueueType.SEND_EMAIL_BOT)
export class SendEmailWorker extends Worker<OutboundEmail> {
	public async run(job: Job<OutboundEmail>): Promise<WorkerRunResponseType> {
		await mail.send(job.data);
		return {
			completed: true
		};
	}
}

At first, I thought I could query the store (hence the queue = Store.from(this)) but that was a bust.
Any ideas on how to accomplish what I am after?
Thanks in advance!

Hello @ITninja04
I see your problem :). Store is a class to attach and retrieve any metadata stored explicitly by a decorator using @StoreSet or @StoreMerge or manually with Store.from itself.
So the decorator implementation should be something like that:
export const WORKER_PROVIDER = "workerProvider";
export const WORKER_QUEUE_TYPE = "workerQueueType";

import {useDecorators, StoreSet} from "@tsed/core";
import {WORKER_PROVIDER, WORKER_QUEUE_TYPE} from "../constants/index.ts";
import {QueueType} from "../types/Queues/QueueTypes";

// isn't necessary to create as specific provider container (but I assume a large part of the Framework use it (debt)).
// registerProvider is enough in most case !
// GlobalProviders.createRegistry("WORKER", Provider, {
//	injectable: true
// });

// export const registerWorker = GlobalProviders.createRegisterFn("WORKER");

export function BullMQWorker(queueType: QueueType): ClassDecorator {
    console.log(associatedQueue);
   return useDecorators(
        // create new provider type and push the provider
       (target: Type) => registerProvider({provide: target, type:  WORKER_PROVIDER}),
       // Then we can use StoreSet to attach our data
       StoreSet(WORKER_QUEUE_TYPE, queueType) // this data can be retrieve later from any module/service
    )
}

Now our decorator is ready, we can consume retrieve all provider with WORKER_PROVIDER type in a factory/module/service.
I suggest to use a Module :)
import {Module, OnReady, InjectorService, Provider} from "@tsed/di";
import {WORKER_PROVIDER, WORKER_QUEUE_TYPE} from "../constants/index.ts";

@Module({
   // you can set default module configuration here but in most case isn't necessary
})
export class BulkMQModule implements OnReady {
   @Inject()
   private injector: InjectorService;

   public getWorkersProviders(): Provider[] {
       return this.injector.getProviders(WORKER_PROVIDER);
   }

   public async $onReady(): Promise<any> {
       this.getWorkersProviders().map(async (provider) => {
          // A store instance is automatically attached to the provider instance
          const queueType = provider.store.get(WORKER_QUEUE_TYPE);
          const serviceWorker = this.injectorService.get<Worker>(provider)!;
          // do something with queueType

          // run worker with the job
          await serviceWorker.run(job)
       })
   }
}

Then import then module with an ES6 import on your server or any file with a bootstrap sequence.
With this example you should have all info to finish your feature ;)
See you
Romain

@Romakita thank you for your response, before I had seen it I was able to address my issue like so:
// Decorator function
export function BullMQWorker(associatedQueue: QueueType): ClassDecorator {
	return (target) => {
		Metadata.set("associated:queue", associatedQueue, target);
		registerWorker({
			provide: target,
			deps: [InjectorService],
			queue: associatedQueue
		});
	};
}

// Worker abstract that all workers have to extend
export abstract class Worker<T> implements IWorker<T>, OnReady {
	logger: Logger = CustomLoggerSetup(new Logger(nameOfClass(this)));
	name = nameOfClass(this);
	queue = Metadata.getOwn("associated:queue", this) as QueueType;
	queueService: QueueService;
	jobOptions: JobsOptions | undefined;
	workerOptions: WorkerOptions | undefined;
	scheduleDelay: undefined | number;
	constructor(options?: QueueOptions) {
		if (options) {
			if (options.jobOptions) {
				this.jobOptions = options.jobOptions;
			}
			if (options.workerOptions) {
				this.workerOptions = options.workerOptions;
			}
			if (options.scheduleOptions && options.scheduleOptions.enabled) {
				this.scheduleDelay = options.scheduleOptions.delay;
			}
		}
	}
	abstract run(job: Job<T>): Promise<WorkerRunResponseType>;
	public testWorker(): boolean {
		return !!this.queueService;
	}
//...more code here
}

My question on using the Metadata method is this, is there a drawback to using the Metadata?

I don't know. I don't use Metadata like you ^^ and I prefer to use Store which is more intuitive and provide an high level API to manage metadata on Class/Method/Prop/Parameter.


I want to skip some epochs

Hello,I am a newbie. When I was training the model, I accidentally closed the process. How could I skip the previous training model?

I ended up at model_epoch13.th7 and I want to continue my model training

I assume the aborted process created some checkpoints? If you re-start the training with the same -savedir (and command-line arguments), it will automatically resume from the last checkpoint.

CUDA_VISIBLE_DEVICES=0,3 fairseq train -sourcelang en -targetlang de -datadir data-bin/news_bpe_2014 -model fconv -nenclayer 15 -nlayer 15 -fconv_nhids 512,512,512,512,512,512,512,512,512,512,768,768,768,2048,2048 -fconv_nlmhids 512,512,512,512,512,512,512,512,512,512,768,768,768,2048,2048 -fconv_kwidths 3,3,3,3,3,3,3,3,3,3,3,3,3,1,1 -fconv_klmwidths 3,3,3,3,3,3,3,3,3,3,3,3,3,1,1 -dropout 0.2 -optim nag -lr 0.25 -clip 0.1 -momentum 0.99 -timeavg -bptt 0 -savedir trainings/final -validbleu -batchsize 48 -maxbatch 1200 &
This is my training command.Should I re-use this order, or do I need to add other parameters?

It should be fine to re-start it exactly like this. After startup, the program should print something like Found existing state, attempting to resume training immediately.

model_best_opt.th7  model_epoch12.th7  model_epoch3.th7  model_epoch7.th7   state_epoch11.th7  state_epoch2.th7  state_epoch6.th7  state_last.th7
model_best.th7      model_epoch13.th7  model_epoch4.th7  model_epoch8.th7   state_epoch12.th7  state_epoch3.th7  state_epoch7.th7
model_epoch10.th7   model_epoch1.th7   model_epoch5.th7  model_epoch9.th7   state_epoch13.th7  state_epoch4.th7  state_epoch8.th7
model_epoch11.th7   model_epoch2.th7   model_epoch6.th7  state_epoch10.th7  state_epoch1.th7   state_epoch5.th7  state_epoch9.th7
This is my saved training model.Thank you very much

Wow, that's great! Thank for your replay.

FYI, the crucial one for resuming is state_last.th7.


Fix: removeAllValues method in totalCost reset

Currently, totalCost is not initialized in removeAllValue. So I added the code and tests.

Great catch - thanks!


Refactor event tracker

Here for the CI.
This PR is composed of two parts:

A refactor and reorganisation of our event tracking system featuring:

A Segment wrapper
A exception-like scheme were each event has it's class, requiring it's own data and always following the same interface_ (initializer, properties and report)

Added 3 more events:

Dashboard visit
Builder visit
Dataset visit

Benefits are:

More compact
It's been extremely easy to add new events.

Downsides:

The  Events module can get messy and organising it into submodules will make it very long to instantiate different events.
All the duplicated report and properties code.

I think benefits outweigh downsides but lets discuss!
Fixes #8962

Could you please CR @juanignaciosl ?
cc/ @jmartinter

I've made some comments, I'll take a second look afterwards. Nice refactor 👌

@juanignaciosl have a second look 😀

I've made some minor comments, up to you taking them into account.
This is a nice refactor and an obvious improvement.
BUT there's something in the overall design that is wrong:

An event is created.
report method is called. It instantiates a SegmentWrapper and calls send_event, which, if Segment is enabled, creates a Resque::EventDeliveryJobs::TrackEvent.
Then, in Resque, TrackEvent, creates the Carto::Tracking::SegmentTracker.new.

As you see, TrackEvent is not actually tracking events but tracking Segment events. TrackEvent is 100% coupled to Segment.
That's why you have a problem when you try to implement null object pattern to avoid constantly checking whether Segment is enabled or not.
I think that this can't be fixed without a bigger refactor that is clearly out of scope. Maybe  the dependency could be made explicit renaming Resque::EventDeliveryJobs::TrackEvent to Resque::EventDeliveryJobs::SegmentEvent and maybe not checking there the configuration. Give it a thought, but don't do any bigger refactor in this PR, which is a nice step forward. I'll be waiting for another CR request if you do any change, but consider merging and deploying if you want.

Thanks for the review, @juanignaciosl
I do see your point about the design. I haven't gone further into redesign since I feel that for that we have to also redesign the events (see for example the duplicated properties being introduced in the wrapper).
I still don't see how you want to make the reported "Static" taking into account that there is now shared instance between rescue or the different machines in the cloud. You'll have to give me a master class on Singleton if it's capable of that 😛 . We should do the same with a lot of the backend code.
Regarding the Segment coupling, I didn't want to over engineer: it's probably easy to change or add more services further down the road.
I've made some minor changes, please review and give a 👍 if you think this is deploy ready for tomorrow morning!

Nah, forget most of my comments xD. Reviewing...

LGTM 👍

These 3 commits:
A typo and a missing .report.
All good in staging :)

👍

@juanignaciosl please have a look at the latest changes!

All events tested in staging without problems now. It's ok for me :+1:

👍 after taking a look at the comment.

Thanks for the comments! Deploying!


Merge in Hugo Docs

Resolves #70

It finally passed on Travis! Merges in Hugo docs, should be an addition only PR. This will add Hugo but does not add links included in the .md document.


Implement new design

basic sketches

you can probably leave out the text at the bottom of those cards. Cards are a link that navigates to:

a standalone site that has its own explanation
a page that has its own explanation

tech notes

ContentCard should definitely be a component
Same with NavMenu


fix(k8s-service): redesign the traffic path to enforce rules in k8s services

New Traffic Path:
Out: Pod -> localGw -> Iptables(kube-proxy) -> localGw -> Policy
In : Gateway -> localGw -> Iptables(kube-proxy) -> localGw -> LocalPods

Codecov Report

Merging #222 (4910bb8) into main (2cb25f2) will decrease coverage by 2.24%.
The diff coverage is 2.51%.

@@            Coverage Diff             @@
##             main     #222      +/-   ##
==========================================
- Coverage   65.20%   62.96%   -2.25%
==========================================
  Files          39       39
  Lines        6125     6334     +209
==========================================
- Hits         3994     3988       -6
- Misses       1787     1993     +206
- Partials      344      353       +9

Impacted Files
Coverage Δ

pkg/agent/cniserver/server.go
0.00% <0.00%> (ø)

pkg/agent/datapath/localBridge.go
37.59% <0.00%> (-19.92%)
:arrow_down:

pkg/agent/datapath/utils.go
36.36% <0.00%> (-20.78%)
:arrow_down:

pkg/agent/proxy/route.go
0.00% <0.00%> (ø)

pkg/utils/utils.go
41.17% <0.00%> (-36.61%)
:arrow_down:

pkg/agent/datapath/multiBridgeDatapath.go
59.34% <100.00%> (+0.24%)
:arrow_up:

plugin/tower/pkg/controller/policy/controller.go
80.39% <100.00%> (-1.92%)
:arrow_down:

pkg/controller/policy/policy_controller.go
74.32% <0.00%> (-0.68%)
:arrow_down:

... and 1 more


String not highlighting for GMS2

Using GMS2 text between quotations does not seem to be highlighting.

Fixed.


Enable requesting LBB companies using a city ID.

This change is

Reviewed 2 of 2 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


Installation failure

Hi, I'm starting to learn Javascript and found your software interesting, and I'm trying to evaluate it. I'm getting errors in the process. The log file is extracted below. Can you help please?
I'm running a Xubuntu 14.04 64bits
Node.js version is the last one 0.12.2
I have already installed sqlite3 version 3.8.2 if it's important.
Many thanks
Log:
20010 silly cache afterAdd<EMAIL_ADDRESS>...
30103 verbose stack Error<EMAIL_ADDRESS>install: node-pre-gyp install --fallback-to-build
30103 verbose stack Exit status 1
30103 verbose stack     at EventEmitter. (/usr/lib/node_modules/npm/lib/utils/lifecycle.js:213:16)
30103 verbose stack     at EventEmitter.emit (events.js:110:17)
30103 verbose stack     at ChildProcess. (/usr/lib/node_modules/npm/lib/utils/spawn.js:14:12)
30103 verbose stack     at ChildProcess.emit (events.js:110:17)
30103 verbose stack     at maybeClose (child_process.js:1015:16)
30103 verbose stack     at Process.ChildProcess._handle.onexit (child_process.js:1087:5)
30104 verbose pkgid<EMAIL_ADDRESS>30105 verbose cwd /home/pvc/sw/jsbin
30106 error Linux 3.13.0-52-generic
30107 error argv "/usr/bin/node" "/usr/bin/npm" "install"
30108 error node v0.12.2
30109 error npm  v2.7.4
30110 error code ELIFECYCLE
30111 error<EMAIL_ADDRESS>install: node-pre-gyp install --fallback-to-build
30111 error Exit status 1
30112 error Failed at the<EMAIL_ADDRESS>install script 'node-pre-gyp install --fallback-to-build'.
30112 error This is most likely a problem with the sqlite3 package,
30112 error not with npm itself.
30112 error Tell the author that this fails on your system:
30112 error     node-pre-gyp install --fallback-to-build
30112 error You can get their info via:
30112 error     npm owner ls sqlite3
30112 error There is likely additional logging output above.
30113 verbose exit [ 1, true ]
30114 verbose unbuild node_modules/sqlite3
30115 info preuninstall<EMAIL_ADDRESS>30116 info uninstall<EMAIL_ADDRESS>30117 verbose unbuild rmStuff<EMAIL_ADDRESS>from /home/pvc/sw/jsbin/node_modules
30118 info postuninstall<EMAIL_ADDRESS>30119 silly gentlyRm /home/pvc/sw/jsbin/node_modules/sqlite3 is being purged from base /home/pvc/sw/jsbin
30120 verbose gentlyRm don't care about contents; nuking /home/pvc/sw/jsbin/node_modules/sqlite3
30121 silly vacuum-fs purging /home/pvc/sw/jsbin/node_modules/sqlite3
30122 silly vacuum-fs quitting because other entries in /home/pvc/sw/jsbin/node_modules

I probably wouldn't recommend using a local jsbin as a way of learning javascript...least of all to say it's hardly the best example of code!
Re: errors. We run on node 0.10.x and whatever version of npm came with it. I doubt if that's it, but certainly give it a try.


Undelivered Mail Returned to Sender for root

I was taking a look at hashbangctl and it caused a security alert to be sent to root, but the mail went undelivered and was returned back to me. I then tried to just send a test email to root and that too was also returned with same error.

error<EMAIL_ADDRESS>mail for hashbang.sh loops back to myself

I think the issue is with /postfix/main.cf
mydestination = $myhostname
I think it actually needs to be
mydestination = $mydomain
I was going to submit a PR on this after making the change, but since I am not 100% sure I figured it best to file as an issue. If this is the case, unless someone fixes it first I can always submit the PR once I get word as I already made the adjustment on my clone of repo. But really with it being such a quick thing, I'm sure anyone can get to it at anytime.
If something more is going on here at least it will open up discussion and maybe I will learn something new.

No, the issue is not with mydestination.
Basically, there are two problems:

there is an issue with sudo(8) that makes it send mail to root whenever someone uses it (it seems to be a PAM snafu, not sure what yet).
there is no<EMAIL_ADDRESS>mailbox, so that mail bounces.

Regarding (2), I plan to deploy mailing lists at some point, but didn't have time yet.

Part (1) fixed by #64.

I was already looking into the list option myself trying to see if /etc/aliases would accept a group rather than usernames (obviously not preferred), but i don't think that is the right approach. I will still look into it and if get before you, at least I cleared one of the many things off your plate

@geoffmcc AFAIK, /etc/aliases doesn't take Unix groups.
Also, the mail routing is done on mail.hashbang.sh, which doesn't use LDAP as its user DB.

PS: @geoffmcc Don't take it to mean you shouldn't propose a solution and implement it, just that the action for that is going to be in docker-postfix, or in a new repo if you pick a new piece of software for the “mailing lists” (or whatever we call those).
The solution I had in mind was running Schleuder - an encrypted mailing-list system - and forward mail to the mailing lists from the postfix container.  Unfortunately, I ENOTIMEd pretty hard lately, and also it looks like Schleuder cannot natively speak LMTP nor could I find a simple/stupid LMTP daemon, so it might involve running a second instance of Postfix to get mail handed over to the container (or find/write a small LMTP implementation).

Part (1) fixed by #64.

I'm not sure how using NOEXEC in sudoers turned off sending mail?
Shouldn't we be modifying one of the mail_* vars?

@daurnimator Removing the attempt to get sudoers data from NSS solved it.
It's just that I did some cleanup in the sudo config while I was at it.
The PR could indeed have a more explicit title.


_hub_core.connect should be renamed to connect_hub

Otherwise it's too confusing with the user-facing function connect()

done


Upgraded Crash to 0.21.1

Codecov Report

Merging #5868 into master will increase coverage by 0.03%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #5868      +/-   ##
============================================
+ Coverage     82.97%   83.01%   +0.03%
- Complexity    12102    12108       +6
============================================
  Files          1426     1426
  Lines         46442    46442
  Branches       5718     5718
============================================
+ Hits          38537    38553      +16
+ Misses         5224     5217       -7
+ Partials       2681     2672       -9

Impacted Files
Coverage Δ
Complexity Δ

...in/java/io/crate/lucene/match/CrateRegexQuery.java
47.82% <0%> (-26.09%)
5% <0%> (-3%)

...io/crate/protocols/postgres/types/VarCharType.java
70.83% <0%> (-20.84%)
9% <0%> (-1%)

...ate/planner/node/dql/PlanWithFetchDescription.java
80% <0%> (-10%)
12% <0%> (-2%)

...rate/planner/consumer/GlobalAggregateConsumer.java
85.71% <0%> (-8.04%)
15% <0%> (-1%)

.../java/io/crate/analyze/symbol/ParameterSymbol.java
82.35% <0%> (-5.89%)
6% <0%> (-1%)

.../java/io/crate/blob/TransportDeleteBlobAction.java
75% <0%> (-5%)
4% <0%> (-1%)

...ration/reference/doc/lucene/IpColumnReference.java
72.72% <0%> (-4.55%)
5% <0%> (+1%)

...in/java/io/crate/breaker/RamAccountingContext.java
81.66% <0%> (-3.34%)
20% <0%> (-2%)

.../operation/collect/sources/ShardCollectSource.java
83.33% <0%> (-2.46%)
29% <0%> (-2%)

...ration/collect/collectors/LuceneBatchIterator.java
87.91% <0%> (-2.2%)
37% <0%> (ø)

... and 31 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 51c7224...48d9a68. Read the comment docs.

Postponing until crash_standalone is fixed (currently not compatible with python2


ask for help

why do you replace ReLU with GuidedBackpropReLU?

This is to implement the combination of GradCAM + Guided backpropagation, as described in the paper.
For plain GradCAM, no need to use this.


Add an option to take screenshots at regular intervals during each test

Is your feature request related to a problem? Please describe.
Videos are made up from screen shots taken after commands are sent, so play very quickly unless slowed. Some ui changes between the commands may also be lost.
Describe the solution you'd like
An option to capture the screen at regular intervals while the test runs.
Describe alternatives you've considered
Proposed implementation at https://github.com/webdriverio-community/wdio-video-reporter/pull/89

Implemented in https://github.com/webdriverio-community/wdio-video-reporter/pull/89


Create a php API and demo page

youtube-dl is often embedded by php applications. There should be an example php page included, with the following features:

Enter URLs and execute youtube-dl (in the background)
Show progress of youtube-dl instances
Offer a way to abort youtube-dl
Allow to download files from the webhost's temporary directory
Offer conversion to mp3/aac/ogg vorbis

This may require an additional output option in youtube-dl, for example the ability to output JSON sentences.

" ;
			}
			?>

@SpEcHiDe Your script is quite dangerous. Imagine the POST data:
submit=Submit&cmqwe=foobar&link=foobar;rm+-rf+/home

Anyway, pass $_POST or $_GET data to shell_exec() without proper escaping is VERY VERY dangerous.

This can be done by mere sanitize the data.I have tried where the index page takes user input url.A session cookie is auto injected.When the user clicks on submit button the ajax requests post the data which is sanitized along with user ip,cookie and store it in mariadb(I have tried in mariadb,so this will also work for mysql or percona db).
ydl_opts and my_hook can be customized to form a json which can be shown to the user as actual download status.
The only issue left here is how to send back user to download the complete file.It was internal so not a big deal.   if d['status'] == 'finished':   to send a callback to download page is what i m thinking now

I have tried the json part https://github.com/siddht1/youtube_dl_embed/blob/master/1.py,this will create a file a.json which can be used as an api feature.Merge this with https://github.com/siddht3/table_json_creator to create a api type.
Things still missing are url provided and extractor details.Which I suppose my_hook has.

It isn't example, but full site: https://github.com/Rudloff/alltube


Running as a Service

Hi,
I'm trying to run PlexTogether as a windows service to run on boot (The web client, not server).  I am trying to configure the service using NSSM, but can't seem to get it working.  Attached is my settings in NSSM, but every time I try to start it, I get an error.  Any idea how to configure this correctly?

I'm not too familiar with NSSM unfortunately.  Does it run ok outside of NSSM? What error do you get when you run it with NSSM?
Ps. I think your arguments for url should be port 8089 instead of 8099 although this won't impact starting the server.


How can we maintain memory or state of chatbot to remember context?

I have made a simple chatbot connected to WhatsApp using Twilio that uses langchain to respond to queries.
The problem is I am unable to maintain the state. I looked at the source code of this repo and here the memory is stored to the disk and then loaded as context.
Is there any other way of doing it?
Thanks.

Hi, what do you mean by unable to maintain the state? Can you describe a scenario?

yeah so for example I asked my bot the following question
Me: How are you my name is Saad.
Bot: I am good. thanks for asking Saad.

Me: Do you remember my name?
Bot: No I don't remember your name.

I haven't exactly deployed your code for my bot, I was only looking at it for inspiration and see how I can maintain the history of conversation so that the bot is more context aware.
This is my code
# Third-party imports
from fastapi import FastAPI, Form, Request
import os
from utils import context_template

#LangChain Imports
from langchain.memory import ConversationBufferMemory
from langchain import OpenAI, LLMChain, PromptTemplate

from twilio.rest import Client

from dotenv import load_dotenv

load_dotenv()

app = FastAPI()

@app.post("/message")
async def reply(request: Request):
    form_data = await request.form()
    print(form_data)

    from_number = form_data['From']
    print(from_number)

    Body = form_data["Body"]

    # Get the Twilio account SID, auth token, and WhatsApp number from environment variables
    openai_api_secret_key = os.getenv("OPENAI_API_KEY")
    account_sid = os.getenv('TWILIO_ACCOUNT_SID')
    auth_token = os.getenv('TWILIO_AUTH_TOKEN')
    whatsapp_number = os.getenv('TWILIO_NUMBER')

    client = Client(account_sid, auth_token)

    #Use langchain Instead of OpenAI
    prompt = PromptTemplate(input_variables=["chat_history", "human_input"], template=context_template())
    memory = ConversationBufferMemory(memory_key="chat_history")

    llm_chain = LLMChain(llm=OpenAI(openai_api_key = openai_api_secret_key), prompt=prompt, verbose=True, memory=memory)

    # The generated text
    chat_response = llm_chain.predict(human_input=Body)
    print(chat_response)

    message = client.messages.create(
            from_=f"whatsapp:{whatsapp_number}",
            body=chat_response,
            to=from_number
    )

    return ""

Sorry long code snippet but if you have any idea please let me know.

That's because in your snippet memory and llm_chain is created newly every time. Try moving definition of prompt memory and llm_chain after app = FastApi()

Ahoy! Thanks, man it worked like a charm.

Another question
If i msg from another number it still uses the context from the previous number I was messaging from.

Yes I was just typing that. it's only ok if you are using it for one user, otherwise you need to store them separately in a dict, database or something based on from_number

Yeah, I am thinking about using Twilio's conversation API or making my custom logic to store and retrieve them.


Ramdisk

First cut at a ramdisk.

There are several issues at the moment:

0) It's read only - just some sample files for debugging at the moment,
ramwrite (if you could call it) would throw an error.
1) ls '#@/subdir' shows the README2 file, but if you try to cat it, it
can't be found
2) Mounting the device removes the 'w' flag from the directory

its'a  reasonable start but you ought to look at devproc.c to see how gen is done.

Is it possible to have this unused in main tree? it would be easy for be patched by all of us in case anyone would discover anything. I mean, using old ramdisk at boot having this.

LGTM
as long as it does not break anything, let's bring it in. It will make your life a lot easier.

let's see what other people think.
No, this won't interfere with an existing ramdisk.


Removing formatting from element after moving cursor then coming back - broken

Hello,
It's pretty basic - moving a cursor after applying a formatting, then going back to remove the formatting does not work and throws errors.
Here's a simple .gif to show:

It does seem to be a pretty major issue since it breaks one of the most basic functionalities.

Short testing showed this error with newer versions of firefox. I'm actually already rewriting the whole inline formatting logic from scratch as it has become erroneous, bloated and on the limit of being un-fixable. Unfortunately I can not give an estimate though when it will be ready and tested.


EvalResult does not clone logged tensors

🐛 Bug

Tensors passed to EvalResult.log() are not cloned by the object, it keeps references to the original tensors. This may become a problem in a somewhat exotic case when the user stores the calculated metric value as a model's property and logs this property as in result.log(self.my_metric).
Then when averaging metrics across batches for the entire epoch, the array of batch metrics contains multiple copies of a reference to the same tensor self.my_metric, the average is therefore incorrect.
This is probably only a problem in the situation so not sure how important it is or whether we want to fix it. A fix like value.clone().detach() could also conflict with the enable_graph option in Result.
To Reproduce
See this colab notebook.

Code sample
class LitClassifier(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.l1 = torch.nn.Linear(28 * 28, 10)
        self.my_metric = torch.tensor(0.0) # User wants to store the current value for some reason
...
    def validation_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.cross_entropy(y_hat, y)
        self.my_metric =  calculate_my_metric()

        result = pl.EvalResult()
        result.log("my_metric", self.my_metric)     # Logged value contains value of self.my_metric in the laste validation step
        result.log("my_metric_detached", self.my_metric.clone().detach())  # Logged value contains average of values produced each step
        return result

Expected behavior
The EvalResult object clones the tensor passed to result.log()
Environment
Colab:

CUDA:

GPU:
available:         False
version:           10.1

Packages:

numpy:             1.18.5
pyTorch_debug:     False
pyTorch_version:   1.6.0+cu101
pytorch-lightning: 0.9.1rc4
tqdm:              4.41.1

System:

OS:                Linux
architecture:

64bit

processor:         x86_64
python:            3.6.9
version:           #1 SMP Thu Jul 23 08:00:38 PDT 2020

Additional context

Thanks for reporting.
@williamFalcon do we need to clone everything?
This may become expensive with many metrics?

yeah, please clone yourself... we can't really do these types of clones since they can OOM


PSMoveClient C API needed

For someone like me, developing linear algebra in C++ is very difficult. I need to do it in Python or Matlab. For that, I need a C-interface to the PSMove client. If I could, I would simply use the openvr plugin and load the openvr dll directly, but unfortunately openvr doesn't work on Mac (please correct me if you know otherwise).
I'm going to start off writing a C API that exposes the functions I need for controller pose data and nothing more. However, I should make sure that I write it in a way that will prove useful for others. Please chime in if you have any specific wants/needs in a C API or if you can point me to an example C API for a VR motion controller that you think we should emulate.

Yeah, as far as I know it appears that OpenVR isn't ready for primetime in OS X. Additionally exposing the C++ interface in Unity is more complicated than exposing a C one. I've been thinking we should consider re-writting the client api with a C-style interface anyway, but just for Unity sake didn't seem like enough of a reason at the time. I think exposing the API in python pushes this need over the edge.
The ClientPSMoveAPI class should be easy to make a C-style interface since it's only has static methods. Some of the structures it contains will need slightly longer names (ex: ClientPSMoveAPI::Response -> ClientPSMoveAPI_Response).
I think the ClientControllerView and ClientTrackerView can become structs. They aren't doing any fancy stuff in the constructor. All of their methods contained within become simple methods that take the struct in as a first parameter:
void ClientControllerView::ApplyControllerDataFrame(
    const PSMoveProtocol::DeviceOutputDataFrame_ControllerDataPacket *data_frame)
{}

becomes
void client_controller_view_apply_data_frame(
    ClientControllerView *view,
    const void *data_frame)
{}

So should i write a C-style interface that wraps ClientPSMoveAPI (and others as needed) or do you expect we might abandon the C++ API if the C interface provides all the necessary functionality? If it's the latter then I will write the C API with the intention of it replacing the C++ API.
Presently PSMoveConfigTool makes use of ClientPSMoveAPI, but also other client source files, and it has knowledge of the protocol. Is it possible for PSMoveConfigTool to be entirely dependent on a C-style interface?
What about the OpenVR driver?
Is there anything else currently using the client?

I think we should abandon the C++ interface long term. However since the configtool, the openvr interface, and the client test tool all use the C++ interface it might be a bit of an undertaking converting everything at once. The actual conversion should be pretty mechanical, but it'll be a lot of code to slog through. If you didn't feel like ripping the band-aid off all at once, so to speak, we could wrap the c++ interface. Then replace stuff in pieces. Then get rid off the c++ interface. If you just wanted to write the wrapper so that it can unblock your work I can do the clean-up later.

I'll write the C interface as though ClientPSMoveAPI.cpp/h doesn't exist.
As the new PSMoveClient.h will be used in other applications, in different languages, it should not include any headers itself. So all the data types will have to be standard C99. I don't know if this'll be a problem but I'll let you handle sharing video frames to clients.
What about the server/client calls? Can we make it even simpler for the client-user? Is there a reason the client should do things like poll_next_message, update, request/response/callbacks? Or is it safe to automate some of that and hide it away in PSMoveClient.cpp?

We could make blocking versions of the calls that wrap the asynchronous ones. So for example the following async call:
t_request_id ClientPSMoveAPI::start_controller_data_stream(ClientControllerView *view, unsigned int data_stream_flags);

could have a blocking version implemented like so:
ClientPSMoveAPI::ResponseMessage ClientPSMoveAPI::start_controller_data_stream_blocking(ClientControllerView *view, unsigned int data_stream_flags)
{
    bool bReceived= false;
    ClientPSMoveAPI::ResponseMessage out_response;

    register_callback(
        start_controller_data_stream(view, data_stream_flags),
        // C++11 lambda that receives callback when response arrived
        // &bReceived and &out_response are local variables that the lambda can change
        [&bReceived, &out_response](const ClientPSMoveAPI::ResponseMessage response, void *userdata) mutable {
            bReceived= false;
            out_response= *response;
        },
        nullptr);

    while (!bReceived)
    {
        sleep(10);
        ClientPSMoveAPI::update();
    }

    return out_response;
}

This is basically the approach libusb uses for it's blocking calls. A client application using this approach will stall as long as the message takes to come from the service. This is fine for command line applications, but not fine for SteamVR or games.
Also we still need to the poll_next_message() event loop for things like controller/tracker list change notifications.

I'm trying to get away from requiring the client-user to write functions to handle messages, of different payload types, of different response types, etc.., then copy data to a local controller state. They don't care about the messaging protocol, they just want the latest controller state.
Without TOO much change to the current architecture, here's what I'm thinking...
[I have to go home now, I'm already quite late, but this message is incomplete. Anyway, take a read and maybe you can head me off before I go further in the wrong direction.]

PSMoveClient::m_implementation_ptr already has m_controller_view_map; maybe the CAPI can simply return a copy of its entries whenever controller state is requested.

And we'll have to continuously issue requests to keep it updated.
How lagged do we expect this to be? Eventually with the state model we can predict pose but button presses may suffer.

For things like get_controller_list(), get_tracker_list(), start_controller_stream(), etc., that can have delays of hundreds of msec without affecting the client much, we can do something like below.

The main problem I'm having is that, without callbacks, update() and poll_next_message() clobber all the messages, so I can't selectively find the message I'm looking for, even if it's a bit old.
PSMResult PSM_GetControllerList(PSMController** controllers, unsigned int timeout_msec). This calls a common function to do the following:

Start the timer
Look to see if there's a pending message of the same type.

If yes, retrieve the message and fire a new request. return.

Look to see if there's a pending request of the same type.

If not, issue the request.

while (no message && timer < timeout_msec)

sleep a tick.
Search for a pending message of the same type.

Return PSMResult_Success or PSMResult_Timeout, message is copied to message pointer argument.

Then the outer function handles populating the list of controllers from the message.

Am I right in my understanding that once a PSMoveProtocol::Request_RequestType_START_CONTROLLER_DATA_STREAM request is sent that the client will monitor the UDP port and automatically ApplyControllerDataFrame? If so then there's nothing I have to do to keep the controller state updated but I have to provide convenience functions for getting the controller state, correct?
The setters are pretty easy, unless we expect the client will want to be informed if a setLED or setRumble failed, after the fact.
Controller list, tracker list, etc... are more difficult. How should I keep those updated (but at a low rate) without requiring the user to write callbacks and registering them with requests? Maybe when I initialize the client I can spin up a thread that takes care of this for the user. What I'm not clear on here is how update() and poll_next_message() would work in this other thread. Any ideas?

Am I right in my understanding that once a PSMoveProtocol::Request_RequestType_START_CONTROLLER_DATA_STREAM request is sent that the client will monitor the UDP port and automatically ApplyControllerDataFrame?

You just beat me to the the reply :). Yes that is the case.

The setters are pretty easy, unless we expect the client will want to be informed if a setLED or setRumble failed, after the fact.

Yeah I don't think we need to do that. I actually want to make the client keep re-publishing to the server until the server converges to the desired state the client wants, but I haven't close that loop yet. There is a question of how to handle authority when you have multiple clients fighting over the same controller that I haven't settled on a best approach yet. So for now you just assume that the set worked.

Controller list, tracker list, etc... are more difficult. How should I keep those updated (but at a low rate) without requiring the user to write callbacks and registering them with requests? Maybe when I initialize the client I can spin up a thread that takes care of this for the user. What I'm not clear on here is how update() and poll_next_message() would work in this other thread. Any ideas?

I think you had the right idea with the timed out PSM_GetControllerList call. If we don't want to deal with a callbacks then we have to block until a reply is received. But this also means that there can only be one pending request in flight at a time so you don't have to worry about dropping other responses on the floor. However, It would currently mean that event notifications currently get eaten but it doesn't have to be that way. For notifications, we could instead set a global "controller/tracker list dirty" flag that gets clear when you ask the ClientPSMoveAPI if a list is dirty. This would remove the need to have a Message type for the notifications,
The PSM_GetControllerList call would be exactly as you had written except that you would need to put a call to ClientPSMoveAPI::update() in the while loop so that you keep polling the sockets for messages (TCP and UDP messages are always fetched on the main thread).

What does ClientControllerView's ListenerCount do? I understand that it's incremented whenever allocate_controller_view is called. In that function, we have the following:
// Keep track of how many clients are listening to this view
view->IncListenerCount();

Maybe I'm thinking about it differently. In my mind, the 'client' is the application using the API, or equivalently the singleton ClientPSMoveAPIImpl that gets spun up during ClientPSMoveAPI::startup().  In either case, the process only has ONE client.
An object that belongs to the client, such as an instance of ClientControllerView, has no knowledge of any other clients on the system.
Would it be more correct if the above comment was,
"Keep track of how many objects are registered to listen in on this controller" ?
So I can spawn 20 different objects in the scene, each of them listening to data from the same controller, and that would increment ListenerCount to +20. Correct?
But if I spin up another process that's also using the API, and it has a single object that's listening to the controller (e.g., a data logger), then that would have a ListenerCount of 1, and the above would stay at 20, correct?

Ah yeah that's definitely a misleading comment. It would definitely be more accurate to say: "Keeps track of how many objects are registered to listen in on this controller on this client" as you mentioned. It's just there to know when to actually deallocate the client controller view when free_controller_view is called (since a client could have called allocate_controller_view multiple times).

During free_controller_view, the m_controller_view_map entry is erased if the listener count is <= 0, as you stated above. Should we also stop_controller_data_stream for this view before it is erased?

Yeah there is no point in sending data to the client if it's going to get dropped on the floor. And if the data stream wasn't running in the first place and we call stop_controller_data_stream anyway, the server already knows the stream isn't running so it will just respond back saying "nope", which is fine.

The only downside I see is if a client wants to frequently drop a view then acquire a new view, then they'll have to async restart the stream (if the drop causes the count to go to 0). I guess this is OK. If someone wants to do lots of drops and creates then they can just hold on to a 'master' view somewhere.

I filled out the header a little and fixed some weird build problems I was having. I'm back to the implementation of the CAPI. I'm still stuck on getting the controller list without a callback.
The method I outlined above requires me to be able to search through ClientPSMoveAPIImpl's m_request_manager's m_implementation_ptr's m_pending_requests. I could write some layered convenience functions in the C++ API to do that, but I'd rather get the simplest solution working first so I can get back to the Kalman filter.
The lambda you implemented above didn't work for me. Can you take a look? You can find it here.

It looks like the lambda function isn't setting the received flag to true:
https://github.com/cboulay/PSMoveService/blob/client_capi/src/psmoveclient/PSMoveClient_CAPI.cpp#L53
If that gets set it looks like that should work. Also I learned recently that you don't need the mutable keyword if the capture parameters are passed in the the '&' prefix. You only need it if you are passing by value and want to modify the capture value in the lambda. So I think what you want is this:
    ClientPSMoveAPI::register_callback(
                      ClientPSMoveAPI::get_controller_list(),
                      // C++11 lambda that receives callback when response arrived
                      // &bReceived and &out_response are local variables that the lambda can change
                      [&bReceived, &out_response](const ClientPSMoveAPI::ResponseMessage *response, void *userdata) {
                          bReceived= true;
                          out_response= *response;
                      },
                      nullptr);

Response Handler: Now with 100% more compilation
PSMResult PSM_GetControllerList(PSMController** controllers, unsigned int timeout_msec)
{
    struct ResultState
    {
        bool bReceived= false;
        ClientPSMoveAPI::ResponseMessage out_response;

        static void response_callback(const ClientPSMoveAPI::ResponseMessage *response, void *userdata)
        {
            ResultState *result= reinterpret_cast<ResultState *>(userdata);

            result->out_response= *response;
            result->bReceived= true;
        }
    };
    ResultState resultState;

    ClientPSMoveAPI::register_callback(
                      ClientPSMoveAPI::get_controller_list(),
                      ResultState::response_callback,
                      &resultState);

    while (!resultState.bReceived)
    {
//        sleep(10);
        ClientPSMoveAPI::update();
    }

    //TODO: handle out_response

    return PSMResult_Error;
}

I have a very basic CAPI working. It's basically just wrapping some of the C++ functions. I've only wrapped enough to get the PSMove controller state. I'll try creating a Matlab wrapper around that.

I had to make a few small cmake build script fixes to build in Windows again:

1477bb3b831b2181fe2d5d045d335d550e7a10a4 - -Wno-switch not a valid MSVC switch
11012f15c3aca17f179dbbdd7aec2d0875f05868 - Boost date_time lib missing in ConfigTool
0f06c719f295e848d86d84004228dba47607252b - PSMoveClient_CAPI install targets

I also got the start_controller_data_stream request to return initial controller state on response. As part of that change I made explicit blocking and async versions of the following calls:

PSM_StartControllerDataStream / PSM_StartControllerDataStreamAsync
PSM_StopControllerDataStream / PSM_StopControllerDataStreamAsync
PSM_SetControllerLEDColor / PSM_SetControllerLEDColorAsync
PSM_ResetControllerPose / PSM_ResetControllerPoseAsync

All the async versions have an optional out parameter for the request id. However they won't be really useful until we have a way to poll messages. I'll take care of that when I use the async methods for the client config tool and steamvr plugin.
Also I added internal blockUntilResponse() helper method to wrap blocking on response since the blocking loop is the same for all blocking calls when you aren't doing anything more than checking for the result enum (which is not the case for get controller list).

Also I was able to run the test_console_CAPI and it looked like it was able to steam data from two controller simultaneously without issue. Thanks for your working on this refactor btw. Looking over the code it this branch it became apparent to me how much of a non-trivial under taking it is. The new client API looks super clean!

Thanks a lot for your work on this.
I'm still not overly happy with requiring the client to (PSMController **)malloc(sizeof(PSMController *)*PSMOVESERVICE_MAX_CONTROLLER_COUNT); before calling PSM_GetControllerList. Working with PtrPtrs in other languages can be a bit difficult, though not impossible. The client also has to remember to free that ** later, assuming no excepting happens in between.
I'm not sure what the alternative design pattern would be. I'm OK with requiring the client to first get a list of controller_ids then to iterate through those to get the PSMController *.
std::array<int, PSMOVESERVICE_MAX_CONTROLLER_COUNT> controller_ids{};
int controller_count = PSM_GetControllerIds(controller_ids.data());
std::vector<std::shared_ptr <PSMController>> myControllers;
for (int ctrl_ix = 0; ctrl_ix < controller_count; ++ctrl_ix)
{
    //if I want this controller
    myControllers.push_back(std::make_shared<PSMController>(get_shared_pointer_to_controller_from_CAPI_singleton's_controller_map(controller_ids[ctrl_ix])));
}

But I don't think I can get a shared pointer from a C-style interface. Getting raw pointers increases the chance of double-deleting. I guess the only way is to use manual incrementing and decrementing, as you were doing, or is there another way?
I'm pretty sure I'm converging on the same result as you had with the C++ API. I could either build the CAPI painfully slowly, as I am now, or I could have taken the C++ API, ripped the functions out of the custom structs and flattened the PSMoveClient class. The result's going to be nearly identical, but at least this way I don't break the C++ API in the process.

Since were going to rip out the C++ interface as soon as the CAPI is done having it converge on the same result (with the addition of a blocking calls) will make the porting easy at least.
I was thinking of filling in the async interface this weekend (PSM_Update,  PSM_PollNextMessage(), etc) this weekend unless you wanted to do that part.
As for the pointer array thing, I'm always a fan of handles in a situation where you have some system managing fixed size pool of entities and you want to minimize the worry of people deleting your internal state.
How does the following sound:

PSM_Initialize pre-allocates the array of PSMController structs, that are by default in a clear state.

Create a new handle type and an accessor to get at the controller state. Internally the "handle" is just an index into the array of pre-allocated PSMController structs. When using controllers you pass around and store the handles, not the pointers. At a point in the code where you actually need to look at controller state you use the accessor:

typedef int PSMControllerHandle;
PSM_PUBLIC_FUNCTION(PSMController *) PSM_GetController(PSMControllerHandle handle);
PSMController *controller= PSM_GetController(my_stored_handle);

PSM_GetControllerIds would still do the blocking work to get the list connected controllers. When it got the list of controllers back it would set the controller ID and controller type on the internal pre-allocated PSMController list. It would fill the out_handles list with a handle for controller entry that had a valid controller type and return the valid controller count, as it did before.

PSM_PUBLIC_FUNCTION(int) PSM_GetControllerList(PSMControllerHandle* out_handles, size_t max_handles);

All controller functions that previously took a pointer will now take a handle. If the handle refers to a controller that doesn't have a valid controller type, we just drop the request on the floor.

PSM_StartControllerDataStream(PSMControllerHandle handle, unsigned int data_stream_flags);

If you want the internal pool of controller structs to get allocated and released during the lifetime of the program you can make the handle a bit more sophisticated and add a unique_id portion. This lets you test if the handle is stale. But in our case I think we're better off just having a fixed pool of k_max_controllers that is in some valid initial empty state.

You're welcome to work on this as much as you want. I'm going to step away from it over the weekend.
I haven't done much since your last push. Today got filled up with other things. I'll come back to it next week, unless you finished at least enough for me to work on the UKF in Matlab.

I finished filling in the C API this weekend. It now has the full blocking and async API for controllers and trackers. The most relevant change to you is how the controller state is maintained and how the list of controllers is fetched. When you want a list of controller that are connected you do the following:
PSMControllerList controllerList;
PSM_GetControllerList(&controllerList);

This gets you back a list of controller IDs, not controller pointers, where a controller ID is just an index between 0 and max_controllers.
When you want to work with a controller you need to register it as listener as before, but you use the controller ID now:
result = PSM_AllocateControllerListener(controller_id);
result = PSM_FreeControllerListener(controller_id);

Behind the scenes this allocates a ClientControllerView into a global array, if no such view has been allocated yet. It also increments a reference count on a global list of PSMControllers that always exists. When you call FreeControllerListener is decrements the ref count. When it hits zero the ClientControllerView is released and corresponding global PSMController object is cleared out.
To get at the controller object you use the GetController accessor:
PSMController *controller= PSM_GetController(controller_id);

All of the controller functions now take a controller_id rather that a controller pointer. So for example the start controller stream method looks like this:
result = PSM_StartControllerDataStream(controller_id, data_stream_flags);

The nice thing about this change is that the client no longer has to worry about managing controller state memory. They just have to keep track of which controller_ids they are using. If you are going to connect to all active controllers, then the PSM_ControllerList is a nice package for this.
Finally you don't call update on the controller state anymore. Simply calling PSM_Update() will update all controllers and trackers that have an active View allocated (i.e. a non-zero ref count).
The async messaging wrapper API is now complete as well so I can proceed with converting over the old C++ client interface use cases. I'll probably start that once this work is integrated back up into master.
I updated the test_client to use the new API so you can see the update API in action.
Let me know if you have any questions or issues with this new code. I only really tested the blocking API, so it's possible something may be broken in the async wrapper code, but since you're not using that it shouldn't be a problem.

Thanks a lot for all of your work on this.
I decided to use cffi for Python.

I wrote a small script to load the header file and remove some pieces that  cffi didn't like.
To make this easier on myself I moved some things to ClientConstants.h
I also had to move some typedefs around a bit.
I noticed that PSM_Initialize wasn't emitting errors even when the service was disconnected.
The blocking calls, for e.g. PSM_StopControllerDataStream need a timeout. I just tried to stop a stream on a controller that didn't open properly and my python app hung.
I prefaced all the enums with a PSM just to be consistent, and in case anyone did a from _psmoveservice.lib import * then they wouldn't be stomping on any names.
For some reason, PSM_AllocateTrackerListener isn't being exported. I commented it out for now. Can you please check your dll file to see if it's being exported? I had to comment out its declaration before building otherwise cffi would complain when I tried to import the cffi-compiled .so It is not commented in the latest push.

BTW, if you're comfortable with python, then you can switch to the misc/python_capi directory and run python compile_psm.py which will hopefully create a _psmoveclient.c file, then compile that into a python extension (.dll/.so). There will definitely be some problems in that compile_psm.py script because it's set up only for my OSX Debug build system.
Then you can follow the comments near the end of the compile_psm.py script for a quick demo on how to use it from within Python. It's actually really simple, and ipython's tab-completion works well.
If you have time tonight, can you give me a quick breakdown of how we might access accurate frame timestamps for our raw sensor data and our tracker images? We might need to add a timestamp field to PSMRawTrackerData.

I noticed that PSM_Initialize wasn't emitting errors even when the
service was disconnected.

The blocking calls, for e.g. PSM_StopControllerDataStream need a timeout.
I just tried to stop a stream on a controller that didn't open properly and
my python app hung.

Thanks for pointing those out. I'll try and fix those issue tomorrow night.
Getting a windows build of PS4EyeCam this evening for psxdev (who worked on
reverse engineering the ps4 camera) so he can help me debug why the video
stream isn't working.
For some reason, PSM_AllocateTrackerListener isn't being exported. I

commented it out for now. Can you please check your dll file to see if it's
being exported? I had to comment out its declaration before building
otherwise cffi would complain when I tried to import the cffi-compiled .so
It is not commented in the latest push.

Whoops. I changed the signature of that method in the cpp and forgot to
update the header. I checked in a fix with the corrected signature:
PSM_PUBLIC_FUNCTION(PSMResult) PSM_AllocateTrackerListener(PSMTrackerID
tracker_id, PSMClientTrackerInfo *tracker_info);
If you have time tonight, can you give me a quick breakdown of how we might

access accurate frame timestamps for our raw sensor data and our tracker
images? We might need to add a timestamp field to PSMRawTrackerData.

Probably the easiest thing to do would be to add a timestamp to the
DeviceOutputDataFrame's ControllerDataPacket message (see line 447 in
PSMoveProtocol.proto). That get's filled out
in ServerControllerView::generate_controller_data_frame_for_stream(). If we
stored the time the controller was opened on the ServerControllerView and
then subtracted that from std::chrono::high_resolution_clock::now() we
could send out the time the controller has been active as a timestamp in
generate_controller_data_frame_for_stream. I think this is ok to do on
publish because we publish the update packet the same frame that we poll
new data from the controller. What gets sent out over the wire is the
latest raw sensor and raw tracker state read that frame.
On Mon, Jul 11, 2016 at 11:38 PM, Chadwick Boulay<EMAIL_ADDRESS>wrote:

BTW, if you're comfortable with python, then you can switch to the
misc/python_capi directory and run python compile_psm.py which will
hopefully create a _psmoveclient.c file, then compile that into a python
extension (.dll/.so). There will definitely be some problems in that
compile_psm.py script because it's set up only for my OSX Debug build
system.
Then you can follow the comments near the end of the compile_psm.py script
for a quick demo on how to use it from within Python. It's actually really
simple, and ipython's tab-completion works well.
If you have time tonight, can you give me a quick breakdown of how we
might access accurate frame timestamps for our raw sensor data and our
tracker images? We might need to add a timestamp field to
PSMRawTrackerData.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/cboulay/PSMoveService/issues/86#issuecomment-231950378,
or mute the thread
https://github.com/notifications/unsubscribe/AQBA_SBVwNBt9sYwlXiv35kJnZanssn6ks5qUzaAgaJpZM4JCrVi
.

I think this is ok to do on publish because we publish the update packet the same frame that we poll new data from the controller. What gets sent out over the wire is the latest raw sensor and raw tracker state read that frame.

Is there any way we can be more accurate than this?
For the controller data frames, is there any chance the service will miss a frame, or maybe poll multiple frames in one pass, so each of those frames needs to be individually timestamped?
For the camera, if we use 60 fps (until 75 fps artifacts get ironed out) then the latest frame we grab may be <1 msec old or it may be 16.6 msec old if we were unlucky. Any way to know exactly how old the frame is?
Then there's transmission delay. Any way to measure it precisely?
I'm not putting this all on your shoulders, just thinking aloud. The more accurate our timestamps can be then the better the filter will work.

For the controller data frames, is there any chance the service will miss a frame, or maybe poll multiple frames in one pass, so each of those frames needs to be individually timestamped?

Good questions! Yeah it is definitely possible to get multiple controller HID data packets per frame for all the reasons you mentioned.
We actually process these packets individually in ServerControllerView::updateStateAndPredict(). Each ServerControllerView has a m_last_filter_update_timestamp_valid that records the last time updateStateAndPredict() was called. We use this to compute the time difference and divide evenly amongst the update packets.  So for example if we had a time difference of 11ms and two data packets we haven't processed, we'd call update_filters_for_psmove twice with time deltas of 5.5ms. On the client you only get the most reset sensor update, not each HID packet. This was because we originally only cared about using the sensor data for testing if the controller was level during calibration and for the magnetometer samples, which don't really require getting every controller update sent down.

For the camera, if we use 60 fps (until 75 fps artifacts get ironed out) then the latest frame we grab may be <1 msec old or it may be 16.6 msec old if we were unlucky. Any way to know exactly how old the frame is?

The updatePositionEstimation() function, which is called immediately before updateStateAndPredict(), just computes the optical tracking position given the latest frame from the camera, but it doesn't timestamp when a new frame comes in. That should be made into the same sort of update packet style update too. We probably should add a isNewFrameAvailable method on the ServerTrackerView. Then we would know both whether to bother updating the optical pose estimate and there would be a notion of how long it had been since the last new frame. Though I'd have to think about what this means with multiple cameras who are updating at independent frame rates when you are trying to triangulate position.

Then there's transmission delay. Any way to measure it precisely?

I was thinking about this the other day. The usual way I know of doing this is getting a round trip time (RTT) and then dividing by two. You could imagine starting a timer, sending a 'ping' request, waiting for the response, stopping the timer, subtracting the different and dividing by two to get the delay.
But because that request is over TCP the RTT would actually be worse than the UDP update stream due to re transmits and packet ordering guarantees (though I don't know if that's really a problem on a localhost connection).
You could imagine doing this over UDP by forcing the service to publish the as fast as possible (regardless of if the controller data was changing or not) and then looking at the differences in data_frame_last_received_time.
But since the client is on the same machine we could just do the easy thing and put a timestamp on the DeviceOutputDataFrame in publish_device_data_frame() and comparing that to the current time in ClientControllerView::ApplyControllerDataFrame().

On the bus ride in this morning I also realized why PSM_Initialize always returns success. It's because initializing is an async operation too. On the config tool I wait until I get the service connected event back. This means I need to make that block as well on the event. Which in turn reminded me that I hadn't actually added the event flags yet either (connection_status_changed, controller_list_changed, tracker_list_changed). That's kind of a big hole at the moment so I'm going to try and do that now real quick.

I just checked in event flag stuff and made PSM_Initialize blocking (and fail properly if there is no connection available). None of the other API you have been using so far has changed. I just added new stuff.

PSM_Initialize now blocks until connection succeeds or fails
Added PSM_InitializeAsync that just kicks off request to connect (like PSM_Initialize did before)
Added events flags that get set when events are received during PSM_Update() or PSM_PollNextMessage()

bool PSM_GetIsConnected()
bool PSM_HasConnectionStatusChanged() - Connected or disconnected
bool PSM_HasControllerListChanged()
bool PSM_HasTrackerListChanged()

PSM_Update() now polls messages automatically and sets event flags
Added PSM_UpdateNoPollMessages() that does what PSM_Update does before

About transmission delay, I was actually referring to on board processing + BT transmission + OS stuff for controller; on board processing + USB transmission + OS stuff for camera. It is these delays that are necessary for calculating the time-steps for the filter updates in the server.
(Side note: I know I'm using the client now to develop the filter, but in the end it'll only be done on the server)
The server-client network delays + display delays are relevant for the prediction time when requesting the 'future' pose, but not when updating the filter. I expect that our first attempt will be to set this to a constant based on what value 'feels good'. Maybe someday we can even make a guitar-hero-like app where the user has to tap the bulb on a table to a visual beat and the delay from velocity minimum to beat time is the amount of prediction needed. I guess this would have to be in-VR.

About transmission delay, I was actually referring to on board processing + BT transmission + OS stuff for controller
Isn't this just the same as measuring the time between bluetooth/USB data packets received? I would think that the board processing + Bluetooth/USB transmission + OS work is constant (on average) and just acts to time-shift the data by some small fixed amount.

I guess the one problem is that the camera frames are going to be timeshifted a different amount than controller bluetooth data. So the video frame you pressed a button may not line up with the frame we receive the button press is received. I wonder if we could determine this time-shift amount with a blink calibration of some kind. Like tell the bulb to turn on and start a timer and wait for the color to show up in the video stream. That would be:
OS HID API + bluetooth transmit + on board processing [light goes on] + OS stuff for camera + usb video frame read = latency to see bulb light up
If we could tell the controller once to pulse the light on a regular interval, keeping a timer locally we could look for the deviation between when we expect to see the blink and when we do actually see the blink. On the DS4 you can set a controller light on time and off time (so it will pulse the bar for you). Does  PSMove_Req_SetLEDPWMFrequency so something similar or is that much higher frequency for dimming purposes (rather than human perceivable flashes?).

Isn't this just the same as measuring the time between bluetooth/USB data packets received? I would think that the board processing + Bluetooth/USB transmission + OS work is constant (on average) and just acts to time-shift the data by some small fixed amount.

It should be approximately constant other than Windows timing quirks but what is the value? It probably varies system-to-system.
But it's not the time between packets, especially not once Ritesh gets the entire frame transferred in a single packet, because that is limited by the sampling rate of the hardware. Sometimes the most recent packet was just copied to the buffer, sometimes it'll be almost 1/fs old. Depending on the setup, sometimes the request might sit and wait until the data are ready (this is typically with analog-to-digital recording devices).
I would love to know both how long ago the packet was pulled off the sensor (variable) and _how long it takes the packet to go from the device to the service (approximately constant).

I would love to know both how long ago the packet was pulled off the sensor (variable) and how long it takes the packet to go from the device to the service (approximately constant).

I just did a bit of digging this morning about how we might go about this (for the psmove anyway). It seems like if we want to measure both the time it takes a sensor packet to be read on board and the time it takes for a sensor packet to make it from the controller to the PC we'd need custom firmware on the controller. For measuring the time between packet reads we need to find the difference between sample packets on the device (though maybe this is already available in the raw timestamps we get in the HID packets?). For measuring the time it takes the packets to get to the PC we could do the RTT/2 method by adding a "ping" request to the firmware. The PC would start a timer, send the "ping" hid request, stop the timer when we get the response.
I came across this guys page about writing custom firmware for the psmove so that he could use it in a quadcopter:
http://www.eissq.com/ps3_move/files/1e93d333d39a9b0d43a47489503859a2-7.html
And it appears that new firmware can be uploaded without any hardware modifications by putting the controller in "DFU" mode and using a firmware uploader utility:
https://code.google.com/archive/p/moveonpc/wikis/HardwareAndFirmware.wiki
That said after looking briefly at his sample firmware code, it appears he only has communication working over USB and not bluetooth.
But before we try any of that, it's probably worth reaching out to this guy to ask him if he knows what these timings are already. Given that he was trying to make this work with a quad copter I'm betting he also had some filtering he was doing that needed this same timing info.

I also came across this bluetooth latency measurement doc where they sent data in the following way:

TestApp sends data through virtual serial port A to bluetooth adapter A
Bluetooth adapter A transmits data to bluetooth adapter B
TestApp reads data from virtual serial port B from bluetooth adapter B
Reports time difference

In this setup for these bluetooth radios it looks like they are getting latencies between 22ms and 89ms (I may be reading those numbers wrong). So basically between 1-3, 60fps frames.
http://www.lairdtech.com/brandworld/library/Application Note - Bluetooth Latency and Data.pdf

@cboulay FYI I just checked in support for timeouts. All blocking calls now take a timeout parameter:

PSM_Initialize
PSM_GetControllerList
PSM_StartControllerDataStream
PSM_StopControllerDataStream
PSM_SetControllerLEDColor
PSM_ResetControllerPose
PSM_GetTrackerList
PSM_StartTrackerDataStream
PSM_StopTrackerDataStream
PSM_GetHMDTrackingSpaceSettings

I updated test_console_CAPI.cpp to pass timeouts into all blocking functions, but I didn't have time to update the python binding code before I had to run out the door. Let me know if you have any trouble with that.
In other news I tried to reach out to eissq.com guy about the controller latency thing, but the e-mail bounced. His last blog post on that site is from 5+ years ago so I guess that's to be expected.

Thanks! Should we make a timeout.count() < 0 block forever? Is blocking forever a useful feature?
The only other thing I had to do was move the #define line to 'ClientConstants.h', otherwise my parser wouldn't clean it sufficiently to auto-generate the python bindings. I don't know if I mentioned that, but the python bindings are automatic, as long as the header file meets (or can be cleaned to meet) the requirements.

Just a heads-up, I rebased against master so you'll have to do a force pull on this branch.

Should we make a timeout.count() < 0 block forever? Is blocking forever a useful feature?

I could see that being useful for debugging. Right now when you set a break point in in the middle of the blocking call or on the service after the request has been sent you can inadvertently time out the call. Though, I guess it would probably be better to get around this by converting the duration to a poll conter (i.e 1000ms timeout = 100 poll count, assuming a 10ms sleep between each poll). That would make the timeout a bit less accurate, but I'm not sure that matters? Beyond that I can't really think of any other case where an infinite timeout matters.

The only other thing I had to do was move the #define line to 'ClientConstants.h', otherwise my parser wouldn't clean it sufficiently to auto-generate the python bindings. I don't know if I mentioned that, but the python bindings are automatic, as long as the header file meets (or can be cleaned to meet) the requirements.

Ah! That's cool the python bindings auto-gen. I'll keep the constants thing in mind if I have to add another constants to the client api.

RE: Measuring bluetooth latency
I was just looking at the Alex's PSMove hardware notes. I think we might be able to measure the bluetooth latency using either Authentication report or the SRAM read/write report. The authentication challenge/response gives us the "ping" request we need to measure RTT/2.

A few seconds after BT connection is established, the PS3 sends 34 random bytes in Feature report 0xA0 as challenge to the Move. A couple of seconds later, it reads back the 22 byte response in Feature report 0xA1. If the Move did not answer correctly, the PS3 closes the connection and reports error 8012170C. Note that there seems to be only one correct response to each challenge. All legit Move controllers will give the same answer if queried with an identical challenge. This authentication process appears to be implemented directly in the BT module since communication between the STM32 and the BT module does not contain any trace of it.

So basically we on some regular interval we could do the following:

Write HID report 0xA0 with some random challenge payload
Start a timer
Read HID report 0xA1 (sends read request over BT, blocks until report data comes back)
Stop timer
Latency= timer duration / 2

Since the challenge processing is "implemented directly in the BT module" I think this means we are bypassing any latency that would be introduced in the OS of the controller and thus this would be a closer measure of actual bluetooth latency. I'm also hoping that you don't actually have to wait "A couple of seconds later" to actually do the 0xA0 read, but that would still be ok if that was the case since the timer starts when we issue the read, not when we issue the challenge.
If this approach doesn't work for some reason we could instead use the SRAM read/write requests, but we have to be more careful with this since both the calibration and BT host address are stored in there. Also we would be incurring latency of the OS at this point.
I might just try the challenge/response test today real quick since we basically have all the parts we need for this in place.

With the last of the C-API refactor work now in master I'm marking this issue as resolved.


Seperate DB tables for user-edited and not-edited alignments

change DB structure and back-end code for accommodating Seperate DB tables for user-edited and not-edited alignments.

Not relevant as per the updated system design


Tests structure

Codecov Report

Merging #512 (c6ee183) into master (ba89f46) will decrease coverage by 0.67%.
The diff coverage is 100.00%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@            Coverage Diff             @@
##           master     #512      +/-   ##
==========================================
- Coverage   92.51%   91.85%   -0.67%
==========================================
  Files         146      146
  Lines        3622     3622
==========================================
- Hits         3351     3327      -24
- Misses        271      295      +24

Files
Coverage Δ

src/settings.py
90.69% <100.00%> (ø)

... and 3 files with indirect coverage changes


Optional input field validation issue. Forces to fill the fields.

Laravel Version: 5.4.*
PHP Version: 5.6.19
Database Driver & Version: 5.7.11

Description:
Optional input field validation issue. I mentioned website &  facebook_url, google_plus_url, twitter_url, linkedin_url as optional field even though i'm getting the validation error while submitting via ajax.
Payload:
{
   "full_name":"Rajasekar D",
   "resume_headline":"UI Engineer",
   "profile_summary":"Currently I work in a software company - Cognizant. Every day I encounter new challenges, which enhances my knowledge of JavaScript and popular frameworks like jQuery and Angular.js. I am also absolutely in love with HTML5 and CSS3. Even though we spend most of the time together creating beautiful pieces of different UIs, I enjoy learning other technologies such as Node.js, RubyonRails.",
   "dob":{
      "year":2003,
      "month":2,
      "day":4
   },
   "gender":"Male",
   "year_of_experience":3,
   "current_salary":60000,
   "expected_salary":60000,
   "website":"",
   "country":"India",
   "state_id":"1",
   "city":"Chennai",
   "full_address":"No. 16, Alex Street, Ullagram, Chennai, TamilNadu, India",
   "postal_code":600091,
   "facebook_url":"",
   "google_plus_url":"",
   "twitter_url":"",
   "linkedin_url":"",
   "date_of_birth":"2003-02-04"
}

Rules:
  public function rules()
   {
       return [
            'full_name' => 'required',
   		'resume_headline' => 'required',
   		'profile_summary' => 'required|min:30',
   		//'profile_picture' => 'max:1024|image|mimes:jpeg,jpg,png,gif',
   		'date_of_birth' => 'required|before:now-14 year',
   		'gender' => 'required|in:Male,Female',
   		'year_of_experience' => 'required|numeric|min:0',
   		'current_salary' => 'required|numeric|min:60000',
   		'expected_salary' => 'required|numeric|min:60000',
   		//'resume' => 'max:1024|mimes:pdf,doc,docx',
   		'website' => 'url',
   		'country' => 'required|in:India',
   		'state_id' => 'required|exists:states,id',
   		'city' => 'required',
   		'full_address' => 'required',
   		'postal_code' => 'required|regex:/^[1-9]\d{5}$/',
   		'facebook_url' => 'sometimes|regex:/http(s)?:\/\/(www\.)?facebook\.com\/.+/i',
   		'google_plus_url' => 'regex:/http(s)?:\/\/(www\.)?plus\.google\.com\/.+/i',
   		'twitter_url' => 'regex:/http(s)?:\/\/(www\.)?twitter\.com\/.+/i',
   		'linkedin_url' => 'regex:/http(s)?:\/\/([\w]+\.)?linkedin\.com\/.+/i',
       ];
   }

Validation Response:
{
   "website":[
      "The website format is invalid."
   ],
   "facebook_url":[
      "The facebook url format is invalid."
   ],
   "google_plus_url":[
      "The google plus url format is invalid."
   ],
   "twitter_url":[
      "The twitter url format is invalid."
   ],
   "linkedin_url":[
      "The linkedin url format is invalid."
   ]
}

If the fields are optional, then you need to add the 'nullable' rule. 'Sometimes' means that the other validation rules will be run if the array of parameters includes said field. Since you're passing
  "facebook_url":"",
   "google_plus_url":"",
   "twitter_url":"",
   "linkedin_url":"",

then these fields are indeed present, therefore the other validation rules will be run. If the field is optional, add 'nullable':
'facebook_url' => 'sometimes|regex:/http(s)?:\/\/(www\.)?facebook\.com\/.+/i|nullable'

@devcircus Works like a charm :+1:


HV-1868 Revert annotation processor service file to javax package

In versions >= 7, the annotation processor isn't working.
Tested in gradle and maven using this guide:
https://docs.jboss.org/hibernate/stable/validator/reference/en-US/html_single/?v=7.0#validator-annotation-processor
In versions prior to 7, the annotation works as intended in maven/gradle builds and rasies compilation errors when an annotation is improperly used. Gradle imports used:
compileOnly 'org.hibernate.validator:hibernate-validator-annotation-processor:6.2.0.Final'
annotationProcessor 'org.hibernate.validator:hibernate-validator-annotation-processor:6.2.0.Final'
implementation 'org.hibernate.validator:hibernate-validator:6.2.0.Final'
In >=7, the annotation processor doesn't run during compilation. Gradle imports used:
compileOnly 'org.hibernate.validator:hibernate-validator-annotation-processor:7.0.2.Final'
annotationProcessor 'org.hibernate.validator:hibernate-validator-annotation-processor:7.0.2.Final'
implementation 'org.hibernate.validator:hibernate-validator:7.0.2.Final'
I don't believe this is something that needs to be changed to jakarta, since javax.annotation.processing.Processor exists in the compiler, not Java EE/Jakarta EE.
I saw there was an issue closed that describes this, HV-1868. Hopefully I can reuse that one.
https://hibernate.atlassian.net/browse/HV-1868

Can one of the admins add this person to the trusted builders? (reply with: "add to whitelist" or "ok to test")


Make a class to set item visibility based on size of window

Hi Bootstrap,
No wonder! you are doing a great job.
I just want to suggest that
How is it, If we toggle the visibility of any item(tag) based on size change, That might be useful a lot.
Thanks

That already exists:

v3: http://getbootstrap.com/css/#responsive-utilities
v4: http://v4-alpha.getbootstrap.com/layout/responsive-utilities/


Button Clicker: The hint gif for step 3 is not relevant with this step

Describe the bug
Button Clicker: The hint gif for step 3 is not relevant with this step
Steps to reproduce the behavior
1.Navigate to https://arcade.makecode.com/beta--skillmap#beginner
2.Button Clicker----->START
Actual behavior
Button Clicker: The hint gif for step 3 is not relevant with this step. This hint is about to set background image. But this step is to teach how to create a sprite.

Additional context
1.OS: Windows(rs6)
2.arcade version: 1.5.26
3.Microsoft MakeCode version: 6.13.54

I'm not sure I understand.  This step is adding a button sprite to the screen. The background was added in the previous level.
Can you add more info?

Oh, nvm. You mean the anigif.  Got it!

Removed the anigif for next PR


Unable to create windows on macOS High Sierra

When calling glfwCreateWindow() I get the following trace at runtime:
2017-11-19 07:54:13.165413-0800 renderer[14025:228846] [General] ERROR: Setting <GLFWContentView: 0x1006c6430> as the first responder for window <GLFWWindow: 0x10052f1c0>, but it is in a different window ((null))! This would eventually crash when the view is freed. The first responder will be set to nil.
I've tried using the following code before I create my window with no success:
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 3);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 2);
glfwWindowHint(GLFW_OPENGL_FORWARD_COMPAT, GL_TRUE);
glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);

Are you also setting GLFW_CONTEXT_VERSION_MAJOR?

@ghuser404 Yeah.  Github's formatting clipped the first line for some reason.

  Just using a simple hello triangle example for testing: http://forum.devmaster.net/t/simple-triangle-using-opengl-and-glfw/24546

@intrigus Awesome, found a fix in one those tickets.  You can close this.

Actually, getting the following output when I try to use a vertex or fragment shader at glsl version 3.3:
GL version: 2.1 INTEL-10.28.29
Error compiling shader type 35633: 'ERROR: 0:1: '' :  version '330' is not supported
ERROR: 0:3: 'layout' : syntax error: syntax error
'
Program ended with exit code: 1

Isn't it because you're setting OpenGL version 3.2? Try 3.3.

@ghuser404 Actually I got it up and working.  It had to do with me not having a VAO bound for newer Mac oses and nothing to do with GLFW.  Thanks for the help.


Sync main > live

Docs Build status updates of commit 8527dba:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

TerminalDocs/dynamic-profiles.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


fix: Add woke rules in project

Pull request

What this PR does / why we need it
Add woke rules yaml file in project to avoid errors when trying to access it URL
Which issue(s) this PR fixes

N/A
Describe testing done for PR

Additional information or special notes for your reviewer

Codecov Report

Merging #650 (44d6ad1) into main (d89f510) will not change coverage.
Report is 3 commits behind head on main.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #650   +/-   ##
=======================================
  Coverage   85.77%   85.77%
=======================================
  Files          68       68
  Lines        4254     4254
=======================================
  Hits         3649     3649
  Misses        473      473
  Partials      132      132


Download link is broken

http://reactide.io/download.html The big "Download" link doesn't link to anything :(

Yeah go look in #122
This project appears to either be waiting for a massive update or it's dead.


Bug in Silver Table Code

I tried to convert a bronze table to silver where the Label.classification.answer column (from flattened bronze table) looked like this:
 [[{"featureId": "redacted_for_confidentiality", "schemaId": "redacted_for_confidentiality", "title": "Response", "value": "response"}], [{"featureId": "redacted_for_confidentiality", "schemaId": "redacted_for_confidentiality", "title": "Response 2", "value": "response"}], [{"featureId": "redacted_for_confidentiality", "schemaId": "redacted_for_confidentiality", "title": "Response 3", "value": "response"}], [{"featureId": "redacted_for_confidentiality", "schemaId": "redacted_for_confidentiality", "title": "Response 4", "value": "response"}]]
As you can see, it is an array of arrays.
It seemed that the add_json_answers_to_dictionary() method failed to process the flattened label column properly. The method iterated through the array of Label.classifications.answer but the ast literal parsing seemed to get stuck after that. Also I think that since the method was designed to parse JSON strings, it was failing to process the Row.
This modification seemed to fix the method (simply tacked the code on at the end), but further testing is necessary.
#this runs if the literal stuff didn't run and it's not json
    if isinstance(answer, list) and len(answer) == 1:
      answer = answer[0] #puzzled how we get a list of length 1 where the contents is the row
      my_dictionary[title] = answer["title"]
    else:
      my_dictionary[title] = answer

I also believe we need to revamp add_json_answers_to_dictionary() to make it easier to debug, as it consists of some relatively brittle code built around edge cases encountered during the many different JSON scenarios we get back from the Labelbox API due to nesting and such.

I have done further investigation and found an issue with the way the library currently handles the JSON export format for dropdown classifications. Radio classifications seem to work just fine. Further investigation is necessary to identify edge cases, and also while we are at it we can investigate support for multi-classification support (e.g. multiple items selected in a checklist or dropdown).

We are beta testing the fix for this right now --it should be pushed to master within a few business days if tests go well.
Anybody who wants to try out the Release Candidate can build a wheel from the nl/classifications_hotfix branch

This has been fixed in 0.4.0 which is now merged.


Update readme.md

the correct option is "brackets" not "bracket"

you could keep "bracket" in the explanation, but the option.arrayFormat must be "brackets" ;)

Uhm, no: https://github.com/sindresorhus/query-string/blob/b4dee485d7f98034aa757d7c5d17b04ce73d2855/index.js#L22


Using Docker and PostgreSQL: How can I set administrator password in official postgresql image? (But securely, not a living environment variable)

I am using the official PostgreSQL Docker imnage (postgresql:11-alpine) from Docker HUB registry.
I want to create the PostgreSQL administrator password in a safe way.
Currently, this image offer to set it using environment variable:

Using environment variables is not really safe as it can be seen after initialization by anyone having access to the host.
There is another option for setting the password but that only work in docker swarm.
In this image, how can I set the PostgreSQL administrator, securely, and without manual step during initial deployment?Using Docker and PostgreSQL: How can I set administrator password in official postgresql image? (But securely, not a living environment variable)

See further down, under "Docker Secrets" (https://hub.docker.com/_/postgres#docker-secrets):

As an alternative to passing sensitive information via environment variables, _FILE may be appended to some of the previously listed environment variables, causing the initialization script to load the values for those variables from files present in the container.

The added functionality to support Docker Swarm's "secrets" feature is fully generic and usable without a "real" container orchestration system at all.
So what you're looking for is POSTGRES_PASSWORD_FILE with a file provided by your orchestration system (and then later removed).
In the future, these sorts of questions/requests would be more appropriately posted to the Docker Community Forums, the Docker Community Slack, or Stack Overflow.


Quote of the Day getting exception

The tuple is raising an exception if the author isn't parsed correctly into the tuple.

Could you share the exact exception message you got? Also, were you using the English or French version?

I'm not getting one today, but it looked like if the quote of the day didn't have an author then the function would raise an exception when you would try accessing raw_data[1]. It was line 158 that raised it yesterday in the english version.  It works today and I'm not sure how often it'll come up.

From what I'm seeing, all recent quotes have had an author attributed (https://en.wikiquote.org/wiki/Wikiquote:Quote_of_the_day/March_2016), it may have been something like the page's HTML layout changing for one day (though this is unlikely).  If you can, next time you get the exception try entering https://en.wikiquote.org/wiki/Main_Page and saving the entire page's HTML to a file, so that I can have a look.

No problem.  Sorry I forgot to write it down, I'm kinda new to the whole using github thing.  Thanks for making this btw!  I'm using it to teach a class for high schoolers and its a great interface.

:+1:


DeepSeek: Fix Hessian Estimation

SUMMARY:
We were running out of memory using GPTQ on deepseek because our hessian memory estimation code assumes all the decoder layers are the same shape. This is not the case for deepseek. Updating the function to find the largest layer across all the decoders fixed the issue.
TEST PLAN:
import torch
from datasets import load_dataset
from transformers import AutoTokenizer

from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
from llmcompressor.transformers.compression.helpers import calculate_offload_device_map

# select a Mixture of Experts model for quantization
MODEL_ID = "deepseek-ai/DeepSeek-Coder-V2-Instruct"

# adjust based off number of desired GPUs
# if not enough memory is available, some layers will automatically be offlaoded to cpu
device_map = calculate_offload_device_map(
    MODEL_ID,
    reserve_for_hessians=True,
    num_gpus=2,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

model = SparseAutoModelForCausalLM.from_pretrained(
    MODEL_ID, device_map=device_map, torch_dtype=torch.bfloat16, trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

# Select calibration dataset.
DATASET_ID = "HuggingFaceH4/ultrachat_200k"
DATASET_SPLIT = "train_sft"
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))

def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False,
        )
    }

ds = ds.map(preprocess)

# Tokenize inputs.
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )

ds = ds.map(tokenize, remove_columns=ds.column_names)

# define a llmcompressor recipe for W416 quantization
# since the MoE gate layers are sensitive to quantization, we add them to the ignore
# list so they remain at full precision
recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W8A8",
        ignore=["lm_head", "re:.*mlp.gate$"],
        sequential_update=True,
    ),
]

oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    save_compressed=True,
)

LGTM - this was tested on the full deepseekv2 model?

I let it run to completion on the lite model, the full model is really slow so I killed it halfway through


Better doc

What is the range of output?

How to get a result like

As mentioned in #7, this type of result is more accurately fractal noise, which this library can be the basis for. See my joshforisha/fractal-noise-js for functions intended for that.


8.9 DCE cluster mode issues

Hey I am trying to use this chart to install, 8.9 DCE in cluster mode. However I am getting the below errors.
Address in property sonar.cluster.es.hosts is not a valid address: sonarqube-dce-9-sonarqube-dce-search-1, sonarqube-dce-9-sonarqube-dce-search-2
Please ignore the 9 in sonarqube-dce-9-, as I just used it for label, the image used is of 8.9.
@jacek-poreda-sonarsource  @tobias-trabelsi-sonarsource @philippe-perrin-sonarsource

Hi @pavan-kumar222 ,
SonarQube 8.9 DCE did not have k8s support and is not supported by the DCE helm chart. As stated in the README.md of this chart the minimal supported version is 9.1 [1].
Try it with the latest version of SonarQube and it should work :+1:
Hope that helps


Customize close button

Is your feature request related to a problem? Please describe.
Icon can be replaced to be anything, the same goes for the action and cancel buttons. But the close Icon cannot be customized, you cant even change the color for it without changing the theme of the toast
Describe the solution you'd like
Much like the other components Id like it to be replaceable with a custom component
Describe alternatives you've considered
Forking this lib and adding this myself, which I will probably do anyway,

I can take a look at this after the weekend. Feel free to create a PR if you want to speed things up 💪


Add LPKX for Ellipse labeling

Sample and schema-only layer packages for ellipse labeling in Pro.

@topowright and @csmoore  - this is a pair of LPKXs for Ellipse labeling in Pro.
Note that it is stand-alone and not in a feature dataset.
@topowright - might make sense for you to add this FC to your feature dataset.
The Arccade expression is working.

Closing this. I am going to grab the latest one and merge this, plus some arcade label expressions that @topowright and I were working on.


install.ps1 analog of install.sh

Does this duplicate functionality that install.msi offers?
https://github.com/chef/vagrant-omnibus/blob/00c19b5ba0c76c2a1684580d4c0c9db4703f0fd7/lib/vagrant-omnibus/action/install_chef.rb#L77
https://github.com/chef/vagrant-omnibus/blob/00c19b5ba0c76c2a1684580d4c0c9db4703f0fd7/lib/vagrant-omnibus/action/install_chef.rb#L204-L221

Other than this msiexec being commented out, :+1:

@lamont-granquist There is some overlap with install.msi, but this also does pre-release and nightlies and does the installation of the downloaded package.

okay then :+1:

:+1: except for the clean up the md5 error message.
This will come in handy!

refactored to a dynamic module - now exports two functions, Get-ChefProjectMetadata which talks to omnitruck to get the package metadata for the requested project and Install-ChefProject which downloads and installs the requested project.

@ksubrama Wanna take a peek and see if the refactor works for you bootstrap scenarios?

:cry: It's so beautiful...

In looking at the installation of ChefDK on Windows, I found that we have two features of the MSI that are not enabled by default - adding the powershell settings, and the HOME environment.
Should those be enabled by default in this? Could we have an option to enable them? I'd love to consume this in pantry.

@jtimberman I'll take a look at that.


Add ability to check integrity of uploaded object

Hi Folks,
I've been reading https://aws.amazon.com/premiumsupport/knowledge-center/data-integrity-s3/ and I'm wondering whether s3fs could/does already support this functionality?
I've got a situation where I request a file from a data provider and in a separate request I can get the checksum of the file.
What I'd like to do is use s3fs to write the file and verify the integrity of the uploaded object by providing the checksum the data provider gives me, something like:
r = requests.get("a-url", stream=True)
if r.status_code == 200:
    try:
        with s3fs.open(f"<bucket-name>/object.type", "wb", md5checksum="ACHECKSUM") as file_out:
            shutil.copyfileobj(r.raw, file_out)
    except IntegrityError as ex:
        # Tidy up here

I've tried using the .checksum() function, but this doesn't return me the correct checksum, I can download the file and get the correct checksum with hashlib.md5().hexdigest(), so I know it's uploaded fine...
Appreciate any pointers you might have on this!

In gcsfs, this is handled explicitly. See, for example, the nice refactor by @nbren12, grouping the integrity checkers in https://github.com/dask/gcsfs/blob/main/gcsfs/checkers.py . That code could be upstreamed to fsspec and made available to other libraries such as this one.
However, the case there is a bit different: it checks that the hash of the data being sent is equal to the has reported back, rather than a user-provided one. It doesn't seem like a bad idea, though!

I wonder if there is an abstraction suitable for fsspec. Currently the checkers in gcsfs query google-specific http headers and json keys.

I'm not sure about the internals of s3fs but put_object in boto3 takes a ContentMD5 and will fail the request if your resultant file doesn't match the checksum. Guessing that isn't quite as easy to use under the hood here?

s3fs uses (aio)botocore, whcih does support this same call

I wonder if there is an abstraction suitable for fsspec.

The code that takes the uploaded data and forms a checksum would be the same, but how that checksum is used would be different.


Added GVRAssetLoader.loadScene, removed unused code

Added asset loader call to replace the entire scene with the loaded
asset. X3D needs this for the anchor node. Added new capabilities to
GVRModelSceneObject: an initial filename and the ability to load that
asset.
GearVRf-DCO-1.0-Signed-off-by: Nola Donato<EMAIL_ADDRESS>

Needs rebase. Please squash the commits and ideally do not use git merge. Git rebase achieves the same and the result is much cleaner history log.

replaced by #842