✨ Select highest tag when there are sibling tags

This fixes issue #238

Oh I didn't notice tests were not passing on CI. I don't have time today I think. I will definitely try to fix those.

I had to approve CI to run. I'll see any further updates sooner now that I'm watching the PR.

I run the tests locally. They are passing.
Thanks for your review!

I'll take a look later this week.

Well, this looks reasonable to me, though I don't know the code in this repo — my involvement here is largely administrative. So I'd be inclined to optimistically merge, as it could always be refined later.
But I have one question for @dwijnand — Dale, is there prior art for the version number comparison logic, something we could be using instead of adding or own custom code...? (I imagine this wheel has been reinvented many times in many programming languages...)

The exact detail of what's considered high or low actually depends on the impl detail of Ivy or Coursier for somethings like 1.0-A1 vs 1.0-RC1, iirc but sbt implements https://www.scala-sbt.org/1.x/api/sbt/librarymanagement/SemanticSelector.html, which lets you create a semantic versioning selector a la npm version expression, which can say greater than based on SemVer logic, which is usually good enough.

@jprudent want to take a stab at using the API Eugene suggests?

sure! let's change the status of PR to changes required, I'll have a look soon

Hello fellows, I need a bit of advice.
This project has 2 modules:

dynver which contains the "pure" logic and independant of SBT
sbtdynver which contains the nuts and bolts of an Sbt plugin. It depends of dynver.

The class SemanticSelector is provided by librarymanagement-core which is an SBT artifact, and not available in dynver module. To use it I need to introduce a dependency or merge both modules. What would you do ?

Add the dependency to the core.

Hello,
I added the dependency as provided. I also added a few tests.
sbt test ✅

I added the dependency as provided

What's the logic there?

Hello Seth, thanks again for your time and feedback.
I didn't notice that the project was cross compiled. The CI does a sbt +test that runs tests for all scala versions, I didn't know.
I just changed the dependency to pull the correct artifact depending of scala version. Note that scala 3 artifacts are still alpha in the repo
The theory behind the Provided was that since the plugin is running in SBT, it already has the library in its classloader when executed and doesn't need to hard depend on it.
But with the new definition of dependency in my last commit fce4d43, the dependency clashes with the one provided by SBT (tests failed in scripted). So I will reintroduce Provided. The danger with this kind of dependency is that the code is compiled against a given version of it, but at runtime the API may change or the Class can't be found. Since I only use a very small subset of it that looks quite stable (VersionSelector and SemanticSelector) we can expect that will never happen.
I will set the dep as Provided and I will write a scripted test (that I discovered in the process...) in a future commit of this PR.
Thanks again and sorry for all that churn.


New folder structure

After successful, AV Scan instead putting a submission in  topcoder-submission bucket; put
it in the following structure:

PR reviewed. Ready to merge.


Add ability to flag request comments for tool admin attention

This also adds a flag to show comments have been edited on the zoom page for easier identification.

Deployed to https://accounts-dev.wmflabs.org/demo/internal.php


Add support for indeterminate state

It would be nice if the switch supports indeterminate state for checkboxes. May be a property which defines the return value in indeterminate state should make this state selectable.

I'd be curious - would you mind sharing an idea or hint how to do it manually (the tri-state behavior)  so that the click event go through all 3 states, not just 2. I'm implementing this component in wicket and I don' t really know JS so I'd like some hint ;)


valueScale is sometimes not available

Commit: v1.4.2
Seen by @mildewey and myself
Steps to reproduce

Not sure. Maybe mouse on the track quickly after initialization?

Observed behavior
react-dom.development.js:287 Uncaught TypeError: this.valueScale is not a function
    at BarTrack.getMouseOverHtml (HorizontalLine1DPixiTrack.js:102)
    at CombinedTrack.getMouseOverHtml (CombinedTrack.js:416)
    at HiGlassComponent.showHoverMenu (HiGlassComponent.js:4999)
    at HiGlassComponent.mouseMoveHandler (HiGlassComponent.js:4958)
    at HTMLUnknownElement.callCallback (react-dom.development.js:147)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:196)
    at invokeGuardedCallback (react-dom.development.js:250)
    at invokeGuardedCallbackAndCatchFirstError (react-dom.development.js:265)
    at executeDispatch (react-dom.development.js:571)
    at executeDispatchesInOrder (react-dom.development.js:596)
    at executeDispatchesAndRelease (react-dom.development.js:695)
    at executeDispatchesAndReleaseTopLevel (react-dom.development.js:704)
    at forEachAccumulated (react-dom.development.js:676)
    at runEventsInBatch (react-dom.development.js:844)
    at runExtractedEventsInBatch (react-dom.development.js:852)
    at handleTopLevel (react-dom.development.js:5025)
    at batchedUpdates$1 (react-dom.development.js:19916)
    at batchedUpdates (react-dom.development.js:2246)
    at dispatchEvent (react-dom.development.js:5105)

Expected behavior

Shouldn't happen

I can consistently replicate this by aggressively moving my mouse over the tracks while they're loading.


MSCloudIdUtils should not be in this tutorial

Install MSCloudIdUtils
First, this is not an official module. It appears to have no source code control as the project page leads to a random employee/person's personal GitHub.
Second, that module is old, has bugs, and adds unnecessary complexity. There are far less complex ways to acquire a token from a certificate.
https://twitter.com/mysticryuujin/status/1153499433193136129

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0a950d6b-db1f-8338-f80a-a3fde54c4b4c
Version Independent ID: 8256d37f-d6dc-a8bf-6a39-d18419c94ac3
Content: Tutorial Get data using the Azure AD Reporting API with certificates
Content Source: articles/active-directory/reports-monitoring/tutorial-access-api-with-certificates.md
Service: active-directory
Sub-service: report-monitor
GitHub Login: @MarkusVi
Microsoft Alias: markvi

@MysticRyuujin Thanks for the feedback !  I have assigned this issue to content author to investigate and update the document as appropriate.

I have created a backlog item to track this.
#please-close


Run error in ArchLinux

As titled.
I've shifted it to AUR pomota-bin
Error
When I run the program and press the button,I got these errors:
$ pomota
GStreamer element autoaudiosink not found. Please install it

(WebKitWebProcess:230207): GLib-GObject-CRITICAL **: 09:28:23.062: invalid (NULL) pointer instance

(WebKitWebProcess:230207): GLib-GObject-CRITICAL **: 09:28:23.062: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed

I can't find any package about autoaudiosink.
Enveriment
OS: EndeavourOS(Arch Linux Based)
Kernel: 6.1.61-1-lts
DE: Cinnamon
gstreamer and lib32-gstreamer: 1.22.6

Hi, sorry but I've been running Windows for the past 6-7 years. But I think, I can help. Can you reach out to me by whatsapp/facebook? links are here: https://p32929.github.io/

Hi, sorry but I've been running Windows for the past 6-7 years. But I think, I can help. Can you reach out to me by whatsapp/facebook? links are here: https://p32929.github.io/
& also, I don't know who created pomota-bin. its my first time seeing it tbh, but thanks to whoever did it actually

@p32929 Thanks for your reply,maybe it's my PC's errors,it run well now.


Can't add or edit labels for node cause create & update in schema is false

Rancher versions: 2.0 master 2/22
Steps to Reproduce:

Create a custom cluster and try to add a label during create
Edit a node in a custom cluster and try to add a label

Results:  the label field in the node schema is false for create and edit.  Should be true.

Version - 2.0 master 3/2
Verified fixed


support everything in hlint

@ndmitchell: hlint uses ANN pragmas and dead-code-detection doesn't know how to handle those. And then it chokes on e.g. this: https://github.com/ndmitchell/hlint/blob/master/src/HLint.hs#L117. I've never used annotations, so I'm a bit unsure how they work and how this should be handled in dead-code-detection. It seems like you can just annotate top-level identifiers with arbitrary haskell expressions. If so, should they be ignored by this tool? Or is using an identifier in an annotation good enough for it to be considered non-dead code?

Modulo annotations this branch now seems to work with hlint. :) (I've added lazy patterns.)

Annotations can be pulled out by any tool to do anything, e.g. extra information for a static analysis pass. I'd suggest just ignoring the content to annotations to start with. You could certainly imagine a situation where a function was only used in an annotation and that was meaningful, but 95% of the time that would deserve cleanup.

Merging this looks valuable from what I can see - although I can't actually test it right now, but if you say it works with HLint I believe you and look forward to trying it out when I'm back at a computer.

Sounds reasonable to me. I'll merge.

Sorry, I'm still fighting some weird ghc exceptions that suddenly appeared...

@ndmitchell: just released a new version (0.6) to hackage.

Thanks! I ran it over HLint and it found a few small things I was able to tidy up. Nice work.


⚠ Bofrost.at has degraded performance

In 669236d, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1375 ms

Resolved: Bofrost.at performance has improved in 04a6757.


Memory leak or greedy memory usage

If you call gc.collect() at the end of build_groups() function (right before return self.groups statement) this will significantly reduce memory usage. It is better to call gc.collect() manually here because you use data structure with reference cycles. This fix allows wrapping some large models, which was not possible without that fix.

Good point, thanks, Ura.
Added gc.collect


Safer implementation of the uninit_vector function

https://github.com/facebook/winterfell/blob/c99530dec063a0757762af6cbfffec3697fdee88/utils/core/src/lib.rs#L78-L82
Hello, here's a safer implementation that uses std::mem::MaybeUninit to create an uninitialized Vec:
pub fn uninit_vector<T>(length: usize) -> Vec<T> {
     let mut vector = Vec::with_capacity(length);
     unsafe {
         let data_ptr = vector.as_mut_ptr();
         std::ptr::write_bytes(data_ptr, 0, length);
         vector.set_len(length);
     }
     return vector;
 }

This implementation uses std::ptr::write_bytes to initialize each element in the vector. This ensures that all elements in the vector are properly initialized, avoiding potential memory safety issues.

Hi! Thank you for brining this up. If I understood correctly, the above would actually initialize all the allocated memory to zeros, right? If so, what is the advantage of doing this vs. just creating a vector with initialized default elements. For example, something like vec![T::default(); length];?

Hi,
If I understand well, the purpose of uninit_vector is to pre-allocate the vector but initialize the items later. Given that, it does not seem interesting to initialize the items with zeros (moreover, 0 may not be a valid value for the items either).
However, the current implementation is unsound. If you read documentation of set_len, you can see you break the first safety requirement: "The elements at old_len..new_len must be initialized."  Moreover, you can't use res[i] = value to initialize res[i], because this reads res[i] and drops it before writing the new value. Last but not least, if anything wrong happens in the function between set_len and the initialization of the last item and causes the function to return early, the vector will be dropped and will run destructor of uninitialized items. You can notice it if you use an item type with a significant Drop implementation, such as Vec<Vec<u8>>.
For example, this crashes due to heap corruption:
let mut v: Vec<Vec<u8>> = Vec::with_capacity(10);
unsafe { v.set_len(10) };
for i in 0..10 {
  v[i] = vec![i as u8]; // This drops the previous value of `v[i]`, which is not initialized
}

In cases where you simply use a for loop to initialize the items, you should take a look at Vec::resize_with. This does not use unsafe code at all and, on my machine, does not seem slower than methods that use unsafe code:
let mut vec = Vec::new();
let mut i = 0usize;
vec.resize_with(n, || { i += 1; value_depending_on_i });

For places where the above cannot apply easily, here are some things you can do instead:

Use MaybeUninit, and use set_len after the items have been initialized.

let mut vec = Vec::with_capacity(n);
let uninit_vec = vec.spare_capacity_mut();
for i in 0..n {
    uninit_vec[i].write(value);
}
unsafe { vec.set_len(n) };

Use ptr::write, which does not read the uninitialized value, and use set_len after the items have been initialized.

let mut vec: Vec<T> = Vec::with_capacity(n);
for i in 0..n {
    unsafe { ptr::write(vec.as_mut_ptr().add(i), value) };
}
unsafe { vec.set_len(n) };


switched to logger for unknown parameters given to updated method in ModelConfig

When calling updated method on a ModelConfig with unknown parameters given, we would get a lot of prints, which was not intended. Now using logger debug for print statements when unknown params given to updated method in ModelConfig. This was occurring because the PretrainedConfig (HF) equivalent config had more params than what was in the ModelConfig and ModelConfig would just ignore them, however print a message.
Note: We may want to consider also inspecting the dataclass to only pass those params that exist when converting from hf config to fms-hf config.

maybe it should be INFO instead of DEBUG?

or even warn?


mesos 0.27.0 support

Will be realized in future?

https://github.com/jenkinsci/mesos-plugin/pull/186#issuecomment-168652356
If you have a Mesos 0.26 setup available, maybe you can make the change and test it yourself, then gives us some feedback.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:40.265s
[INFO] Finished at: Tue Feb 02 21:14:44 EET 2016
[INFO] Final Memory: 44M/378M
[INFO] ------------------------------------------------------------------------

befor
      if (StringUtils.isNotBlank(secret)) {
        Credential credential = Credential.newBuilder()
                .setPrincipal(principal)
                .setSecret(ByteString.copyFromUtf8(secret))
                .build();

after
      if (StringUtils.isNotBlank(secret)) {
        Credential credential = Credential.newBuilder()
                .setPrincipal(principal)
                .setSecret(secret)
                .build();

But there are several problems - I hope you help:

I try to forward this tutorial

sudo mvn hpi:run
I open `localhost:8080` and see this error
![2016-02-02-220135_1600x900_scrot](https://cloud.githubusercontent.com/assets/13286493/12762603/9799322a-c9f8-11e5-87e7-c874b8b6c86d.png)

I had configured mesos cloud in jenkins settings and create new job - execute shell command
```sh
sleep 3000

but there aren't neither  in jenkins dashbord nor in mesos dashbord nothing new.
Can you help?

http://localhost:8080/jenkins
What does the jenkins logs say ?

Feb 02, 2016 10:00:59 PM INFO jenkins.InitReactorRunner$1 onAttained
Started initialization
Feb 02, 2016 10:00:59 PM INFO jenkins.InitReactorRunner$1 onAttained
Listed all plugins
Feb 02, 2016 10:00:59 PM INFO jenkins.InitReactorRunner$1 onAttained
Prepared all plugins
Feb 02, 2016 10:00:59 PM INFO jenkins.InitReactorRunner$1 onAttained
Started all plugins
Feb 02, 2016 10:00:59 PM INFO jenkins.InitReactorRunner$1 onAttained
Augmented all extensions
Feb 02, 2016 10:01:00 PM INFO jenkins.InitReactorRunner$1 onAttained
Loaded all jobs
Feb 02, 2016 10:01:01 PM INFO org.jenkinsci.main.modules.sshd.SSHD start
Started SSHD at port 60868
Feb 02, 2016 10:01:02 PM INFO jenkins.InitReactorRunner$1 onAttained
Completed initialization
Feb 02, 2016 10:01:02 PM INFO hudson.TcpSlaveAgentListener
JNLP slave agent listener started on TCP port 35783
Feb 02, 2016 10:01:02 PM INFO hudson.WebAppMain$3 run
Jenkins is fully up and running
Feb 02, 2016 10:01:02 PM INFO jenkins.model.Jenkins
Effective SlaveRestarter on master: null
Feb 02, 2016 10:01:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:01:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 33 ms
Feb 02, 2016 10:02:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:02:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 0 ms
Feb 02, 2016 10:03:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:03:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 0 ms
Feb 02, 2016 10:04:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:04:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 1 ms
Feb 02, 2016 10:05:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:05:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 0 ms
Feb 02, 2016 10:06:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:06:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 0 ms
Feb 02, 2016 10:07:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 02, 2016 10:07:08 PM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 68 ms
Feb 03, 2016 6:22:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 03, 2016 6:22:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 1 ms
Feb 03, 2016 6:23:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 03, 2016 6:23:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 0 ms
Feb 03, 2016 6:24:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 03, 2016 6:24:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 0 ms
Feb 03, 2016 6:25:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Started Mesos pending deletion slave cleanup
Feb 03, 2016 6:25:18 AM INFO hudson.model.AsyncPeriodicWork$1 run
Finished Mesos pending deletion slave cleanup. 1 ms

In fact - after managing mesos cloud in jenkins dashboard new framework must be registered in mesos dashboard. But in my case it empty.

Have configured 'on demand registration - yes ?

I had :

installed standalone jenkins
copy plugin

Now messos see jenkins framework.
But I have new trouble:
when I create new job - shell command -sleep 600 and press "to schedule" button - in jenkins dashboard I see that this job processing starts but in mesos dashboard I don't see any new task.

And yet one more question - how to make all builds only on slave instances?

have you labelled the job as 'mesos' ?
Yes, this plugin creates jenkins slave instances when there is a build in queue and kills it when the builds is complete.

I had created new job with mesos label, checkpointed Mesos Single-Use Slave option.

Can you please check the jenkins logs if the resource offers are sent from mesos ?

http://pastebin.com/4rGBS19j

Is there any other framework other than Jenkins registered in your Mesos ?
Can you try restarting your jenkins ?
/restart

I had
sudo service jenkins restart

and recreate job - the same error.

I had also marathon installed - but it's disabled (sudo service marathon stop).

Not sure whats going wrong yet.
I'll squeeze to squeeze sometime this week to debug Mesos 0.26.0 with Jenkins Mesos plugin. Meantime if you have any updates please post it here.

I upgraded to mesos 0.27.0, recompile by sudo mvn package, remove in jenkins plugins path mesos.hpi and mesos dir-ry, restart mesos master and slaves, jenkins services.
this is mesos in jenkins settings:

I use debian apt repo from mesosphere.

Maybe problem in permissions? Marathon run with root permissions, jenkins - with jenkins user permissions.

Above screenshot, I see that resources are offered to Jenkins. Can you check Mesos sandbox logs ?

http://pastebin.com/QjGr7VR7

Have you configured LIBPROCESS_IP & cie variables?
2016-02-03 8:19 GMT+01:00 Alex Ushakov<EMAIL_ADDRESS>
http://pastebin.com/QjGr7VR7
—
Reply to this email directly or view it on GitHub
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-179060001
.

No. In instruction i didnt see that.
Where they should be configured? Or you mean cloud (mesos) settings in
jenkins settings?
On Wed, Feb 3, 2016 at 4:10 PM, geoffroyjabouley<EMAIL_ADDRESS>wrote:

Have you configured LIBPROCESS_IP & cie variables?
2016-02-03 8:19 GMT+01:00 Alex Ushakov<EMAIL_ADDRESS>
http://pastebin.com/QjGr7VR7
—
Reply to this email directly or view it on GitHub
<
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-179060001
.

—
Reply to this email directly or view it on GitHub
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-179256875
.

I want to confirm that i am using jenkins plugin (0.10.1) with mesos 0.27.0 and it is working fine. So i guess we can close this issue.

Hello @pradeepchhetri,
When builduing the plugin, you didn't had this problem:
[ERROR] ...../src/main/java/org/jenkinsci/plugins/mesos/JenkinsScheduler.java:[559,18] cannot find symbol
symbol:   class ContainerInfo
location: class org.apache.mesos.Protos.CommandInfo

@alexxxxx You dont need to build the plugin with each mesos version. You can just install the mesos plugin from Jenkins UI ("Manage Jenkins" -> "Manage Plugins") and install it from there.
Let me know if it works for you or not. I will help you out.

Thanks a lot for your quick reply.
I have build and install the plugin (0.10.1 with mesos 0.26.0). from the
UI, I can connect to mesos.
My problem is that the framework received offers but decline them all (and
I don't know how to see the reason).
This is why I was trying to compile the plugin with the latest mesos (The
one I am currently running)
Cheers,
Alex
On Wed, Feb 24, 2016 at 2:26 PM, Pradeep Chhetri<EMAIL_ADDRESS>wrote:

@alexxxxx https://github.com/alexxxxx You dont need to build the plugin
with each mesos version. You can just install the mesos plugin from Jenkins
UI ("Manage Jenkins" -> "Manage Plugins") and install it from there.
Let me know if it works for you or not. I will help you out.
—
Reply to this email directly or view it on GitHub
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-188253816
.

Hello Alex,
First thing i would like you to check is:
Did you configure Mesos Cloud correctly (specially did you set the Label string and using the same Label in the job configuration as well) ?
I am attaching the screenshot of both below:
Mesos Cloud Setting

Jenkins Job Setting

I confirm that the label is properly set. I also use the magic role *. The
only difference is that I don't put any credentials since my mesos only
authenticate slave.
this is the mesos logs:
I0224 15:13:46.972976 30789 master.cpp:5350] Sending 1 offers to framework
24baf11e-29b4-4175-bfde-e80e3f435036-0026 (Jenkins Scheduler) at
scheduler-ab2279f8-55a2-4c24-a0bd-c00703e95cde@<IP_ADDRESS>:45937
I0224 15:13:46.975461 30791 master.cpp:3673] Processing DECLINE call for
offers: [ 24baf11e-29b4-4175-bfde-e80e3f435036-O3379 ] for framework
24baf11e-29b4-4175-bfde-e80e3f435036-0026 (Jenkins Scheduler) at
scheduler-ab2279f8-55a2-4c24-a0bd-c00703e95cde@<IP_ADDRESS>:45937
And in the jenkins executor list, I see that the plugin kepp trying to
bootstrap new executor.
Alex
On Wed, Feb 24, 2016 at 3:03 PM, Pradeep Chhetri<EMAIL_ADDRESS>wrote:

Hello Alex,
First thing i would like you to check is:
Did you configure Mesos Cloud correctly (specially did you set the Label
string and using the same Label in the job configuration as well) ?
I am attaching the screenshot of both below:
Mesos Cloud Setting
[image: screen shot 2016-02-24 at 13 59 05]
https://cloud.githubusercontent.com/assets/2232667/13287458/208141d8-daff-11e5-8a2e-53c8612f6a83.png
Jenkins Job Setting
[image: screen shot 2016-02-24 at 14 00 52]
https://cloud.githubusercontent.com/assets/2232667/13287474/2dc9ad9e-daff-11e5-9d63-e87751cc9256.png
—
Reply to this email directly or view it on GitHub
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-188267476
.

Hello
have you properly set the field "slave username"? This must be set to the
unix user to be used on the mesos slave machine.
2016-02-24 15:19 GMT+01:00 abeche<EMAIL_ADDRESS>
I confirm that the label is properly set. I also use the magic role *. The
only difference is that I don't put any credentials since my mesos only
authenticate slave.
this is the mesos logs:
I0224 15:13:46.972976 30789 master.cpp:5350] Sending 1 offers to framework
24baf11e-29b4-4175-bfde-e80e3f435036-0026 (Jenkins Scheduler) at
scheduler-ab2279f8-55a2-4c24-a0bd-c00703e95cde@<IP_ADDRESS>:45937
I0224 15:13:46.975461 30791 master.cpp:3673] Processing DECLINE call for
offers: [ 24baf11e-29b4-4175-bfde-e80e3f435036-O3379 ] for framework
24baf11e-29b4-4175-bfde-e80e3f435036-0026 (Jenkins Scheduler) at
scheduler-ab2279f8-55a2-4c24-a0bd-c00703e95cde@<IP_ADDRESS>:45937
And in the jenkins executor list, I see that the plugin kepp trying to
bootstrap new executor.
Alex
On Wed, Feb 24, 2016 at 3:03 PM, Pradeep Chhetri<EMAIL_ADDRESS>
wrote:

Hello Alex,
First thing i would like you to check is:
Did you configure Mesos Cloud correctly (specially did you set the Label
string and using the same Label in the job configuration as well) ?
I am attaching the screenshot of both below:
Mesos Cloud Setting
[image: screen shot 2016-02-24 at 13 59 05]
<
https://cloud.githubusercontent.com/assets/2232667/13287458/208141d8-daff-11e5-8a2e-53c8612f6a83.png
Jenkins Job Setting
[image: screen shot 2016-02-24 at 14 00 52]
<
https://cloud.githubusercontent.com/assets/2232667/13287474/2dc9ad9e-daff-11e5-9d63-e87751cc9256.png
—
Reply to this email directly or view it on GitHub
<
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-188267476
.

—
Reply to this email directly or view it on GitHub
https://github.com/jenkinsci/mesos-plugin/issues/201#issuecomment-188273559
.

Related PR :  https://github.com/jenkinsci/mesos-plugin/pull/210

Do you have more than one label set? I am running this with Mesos 0.27.1 and Marathon 0.15.3 - all good.
Or, perhaps resource offer does not match demand?

I'm running Mesos 0.28.2 and Jenkins 1.651.3 (also testing 2.7) with plugin version 0.13.0, (as I saw that 0.13.1 is tied to Mesos 1.0)
I'm getting disconnected.

my configuration is this:

1.0 and 2.0 will not aim to support 0.27.0.


Remove sudo when it's a root user in zthin layer

Signed-off-by: jichenjc<EMAIL_ADDRESS>

thanks Chuck :)


Crash appears when section is deleted

When I delete sections from tableview I got crash in function:
func performFillAction(action: SwipeAction, fillOption: SwipeExpansionStyle.FillOptions)
line 452 in called: tableView.deleteRows(at: [indexPath], with: .none)
I think that the root case is in fact that at that moment in my code I am deleting section with tableView.deleteSections(indexSet, with: .fade), I've tried the same steps without SwipeCellKit and everything work fine.
Error message:

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid update: invalid number of rows in section 0.  The number of rows contained in an existing section after the update (1) must be equal to the number of rows contained in that section before the update (1), plus or minus the number of rows inserted or deleted from that section (0 inserted, 1 deleted) and plus or minus the number of rows moved into or out of that section (0 moved in, 0 moved out).'

not a bug: solution is to use action.fulfill(with: .delete) instead

instead what?


We should consider promoting "Properties" fields in specs up a level to make for a cleaner spec

For example in the generated Redis types, there is a field Properties which probably ought to have its fields promoted up a level.
This is a pattern that is quite common (for example Storage account also has a Properties)

I thought this quite often while going through the AnyType occurrences. Do you mean specifically for fields called "Properties"? Also, I'd guess we would want this to be recursive just in case there are multiple levels of Properties?

Yeah, specifically the properties field on the spec.
Not sure about nested properties. That first properties field is actually an artifact of ARM (everything outside of it is "for ARM", everything inside of it is "for the RP/Service") - it's a confusing distinction for actual consumers of the API though and AutoRest supports field collapsing to squash the contents of it up into the top level object. Removing that reduces nesting and I think all ARM resources have it.
Nested properties are a different story as those may actually be in the service API?

I think we can automatically detect what should be processed as some of the types are marked as flatten in the Swagger specs. (But this takes us further away from the ARM JSON…)

Ah ok - definitely for the properties on a spec particularly.


Bug Report: The Engine Spams Guide Button (KE 1.8)

This Bug Makes The Game Unplayable
Here's The Video: https://youtu.be/7sNAWjQzTxA

nevermind, i just forgot to disconnecting my ps2 controller

nevermind, i just forgot to disconnecting my ps2 controller


Build is always executed if mtimes are missing

Today a user handled me a rar archive (from a windows machine) with a Cargo package. When I extracted it on my linux machine, I've noticed that cargo build always rebuilds the project, instead of doing zero builds. Looks like the problem is that extracted files miss mtimes, so we always rebulid in here:
https://github.com/rust-lang/cargo/blob/f46d485b8d149b06aeab62b207e9a8f2c1015aea/src/cargo/ops/cargo_rustc/fingerprint.rs#L250-L252
Seems like a niche use case, and I am not sure how to fix it properly (touching files seems inappropriate?), but lets log the issue anyways!

Hm I'm not actually sure how an mtime can be missing... Did the archive include the target directory with fingerprints?

Hm, that's super weird. I've tried to reproduce it now, and I can't =/ I would say it was a hallucination, if not for the fact that I happen to have one of terminal tabs with experiments laying around, which clearly shows rebuild happening: https://gist.github.com/matklad/0fe6554ecf473b3af9e511e197921614

Did the archive include the target directory with fingerprints?

Yeah, but I think I've tried removing the target directory, and rebuilds still happened, but given that I can't reproduce even the original issue, I am not sure in anything now...

Triage:  This issue is quite old, and seemingly hard to reproduce.  There's a lot of discussion (some even recent!) in #6529 about moving away from mtimes and to try things like checking hashes.
Given this issue here doesn't seem particularly actionable, it probably makes the most sense to just close it.

Yea, without a clear reproduction this probably isn't actionable.
Also, the fingerprint code has changed a lot since this was written, and it may be possible that it no longer behaves the same.


Select only newFile without extension

I would like that on a creation of a new file only the newFile Part of the new filename is selected. Because of that it is possible to directly write the new name and don't rewrite the extension.

Hi @DerAlbertCom,
From what I see, there is currently no way to do this. Will close this issue for now, but I'll keep my eyes open.
Cheers,
Dominik

I'd just like to point out that this issue will be fixed by https://github.com/dkundel/vscode-new-file/pull/27 !


only 6 buttons can be pressed at the same time

There is a problem: it is impossible to press more than 6 buttons at the same time. Most likely, this is related to the length of the array that stores what is currently activated. The length of this array is 6 elements as buttons on linked
controller and possibly increasing this length to 25 (the number of buttons on your controller) the problem will disappear.
P.S. translated by Google translator

This part of the code does not use an array (in opposition to create's linked controller implementation, which does uses an array). Instead, I use a 16 bit bitmask to send the state of all buttons, and aother 32 bit variable to store the analog axis values. Here is the relevant code: ControllerRedstoneOutput
I tested in a world just in case, I can press simultaneously all buttons of the controller without any issues.
What are you using as input source, a keyboard/mouse or a gamepad ?
If you are using a keyboard, then I think the issue is that your keyboard can't register more that 6 keys being pressed at the same time. In this case there is nothing that I can do since its a hardware issue.
You can check this using the following website : https://www.mechanical-keyboard.org/key-rollover-test/

Is there anything new about your problem ?

Closing this for now, let me know when you have anything new on this and I will reopen it


Add Package Auto-Discovery

Hi!
Consider this a draft and just a POC, feel free to close it right away.
I got this minor issue when first trying the package:

And quickly got it working by adding the provider in app.php.
Thanks!

Good catch @fgilio, thank you!


Upgrade golang version to 1.8

Also upgraded:
github.com/cert-manager/cert-manager v1.8.0 => v1.8.1
github.com/container-storage-interface/spec v1.4.0 => v1.6.0
github.com/go-logr/logr v1.2.0 => v1.2.3
github.com/kubernetes-csi/csi-lib-utils v0.9.1 => v0.11.0
golang.org/x/net v0.0.0-20220107192237-5cfca573fb4d => v0.0.0-20220615171555-694bf12d69de
google.golang.org/grpc v1.43.0 => v1.47.0
Signed-off-by: Jing Liu<EMAIL_ADDRESS>

Thanks!
/approve
/lgtm
/ok-to-test


Resolve new cypress failures

Description
Resolve new cypress failures from feature anywhere changes
Issues Resolved
N/A
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Cypress test failures now match the main branch when executing yarn run cypress open and executing all the tests


Lowering required CMake version

JerryScript requires CMake 2.8.12.
cmake_minimum_required (VERSION 2.8.12)

However, It seems like some version of ubuntu only support CMake 2.8.7 with 'apt-get'  that developers running such ubuntu should manually download and build CMake 2.8.12.
Is there any problem lowering minimum required version of CMake to 2.8? If there are no problems why don't we lower the required version.

On ubuntu 12.04 which has CMake 2.8.7 the gcc is also old. It's only 4.6 which can not compile the engine (AFAIK). So on that systems the user/developer should also install a new gcc. I think this is a bit more general question on what is the minimum required version of tools for which we support the build.
I'm not against modifying the required version for the cmake, I'm just concerned that on such systems this step is not enough to build the engine.

I think latest Ubuntu LTS should be fine, so we may just make it default for developers.

I found that we could not build jerry without changing CMake script since we use target_include_directories, target_compile_difinitions and maybe more which are added from CMake 2.8.11. That my conclusion is that ubuntu 12.04 is not sufficient to build Jerry because of CMake version problem as well as gcc and probably other infrastructure.
Closing this issue for won't fix.


Clarify and implement leftover methods in TSD

@LorenzRychener This is done in the work I made for #10 I'll do a PR soon :-)


Window for resolving unmerged changes is empty

Current Behavior
We did some changes on the same nodes on two different branches.
Once we tried to pull second branch to the first one, node-red shown that there are some conflicts.
Window for resolving conflicts keep empty.
In browser debug console is written that the JSON is not valid.
I expect that the GIT does some changes on JSON file, which is not valid anymore - checked updated flows.js file by validator.
In other words:
After pulling different branch to current branch the window for resolving unmerged changes is empty.
On browser debug console is written:SyntaxError: Unexpected token '<', ..."9b79298", <<<<<<<<< "... is not valid JSON     at JSON.parse (<anonymous>)     at Array.<anonymous> (red.min.js?v=3.0.2:18:157662)     at c (vendor.js?v=3.0.2:2:28294)     at Object.fireWith [as resolveWith] (vendor.js?v=3.0.2:2:29039)     at vendor.js?v=3.0.2:2:31222     at c (vendor.js?v=3.0.2:2:28294)     at Object.fireWith [as resolveWith] (vendor.js?v=3.0.2:2:29039)     at s.<computed> [as resolve] (vendor.js?v=3.0.2:2:30940)     at Object.<anonymous> (red.min.js?v=3.0.2:18:157535)     at e (vendor.js?v=3.0.2:2:30005)

Expected Behavior
I expect that the tool will show changes and provide selector to choose the correct change and let me resolve the merge conflict, as it usually do.
Steps To Reproduce
I dont know how to reproduce.
Usualy it is working well.
Example flow
Part of affected flow.json:
`  {
"id": "024bad5d3b76bc32",
"type": "junction",
"z": "821581ae4bd94b3c",
"x": 1320,
"y": 580,
"wires": [
[
"ea7918c97050254e"
]
]
},
{
<<<<<<< HEAD
"id": "a7d8924f4b27f0ab",
"type": "junction",
"z": "b46f638bc028ac63",
"g": "7cfdb8db608dfb49",
"x": 2120,
"y": 900,
"wires": [
[
"2037a6707ca47d80",
"467535432b43ead6"
    "id": "77b3bd088cf68a8f",
    "type": "junction",
    "z": "2098bdc4a5f94ecf",
    "g": "19112c4a39466b58",
    "x": 1280,
    "y": 2260,
    "wires": [
        [
            "f266235c080d335d"
        ]
    ]
},
{
    "id": "b8abba8cfc77d332",
    "type": "junction",
    "z": "2098bdc4a5f94ecf",
    "g": "19112c4a39466b58",
    "x": 1300,
    "y": 2060,
    "wires": [
        [
            "13061ad02debc634"

588205d851159a8bb484e17c417bdf02c1cf434b
]
]
},
{
"id": "3614757aa34b6c6e",
"type": "ui_base",
"theme": {
"name": "theme-light",
"lightTheme": {
"default": "#0094CE",
"baseColor": "#0094CE",
"baseFont": "-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen-Sans,Ubuntu,Cantarell,Helvetica Neue,sans-serif",
"edited": true,
"reset": false
},`

Environment

Node-RED version: v3.0.2
Node.js version: v18.12.0
npm version: v8.19.2
Platform/OS: Windows 11
Browser:All

As you say, it looks like conflicts have been merged into the file without being properly resolved - so it contains invalid JSON. This is beyond the ability of Node-RED to resolve. You will have to fall back to the git command line to revert the merge and get back to clean JSON. The project repositories are located under the projects directory in the .node-red directory.

Thank you for a quick response.
And my expectation that this conflict resolver a part of Node-red is not able to handle this kind of git behavior?
It happened many times, that resolver was able to show the problems and provided selector to choose which change should be applied.
Why it not worked this time?
The picture what I am expecting help more:

Why it not worked this time?

As you say, this normally works. You must be hitting some edge case that we've not seen before. I cannot say why it has failed this time. I'm also not sure what information you can usefully provide to help us try to recreate.


Publishing to Open VSX Registry (for VSCodium, etc.)

Hi,
your extension looks good. :+1:
Would you mind to publish it also to the Open VSX Registry? Then it can also easily be added as extension in VSCodium, Ecplise Theia and other free IDEs (see this Eclipse newsletter).
Thanks.

Hello, @Dominik-K, the extension is for Visual Studio. Does it work for VSCode or exlipse out of the box?

Oh, right. Damn, I've overseen this. Sorry.


i love this crab game mod

i love it please keep updating this mod you are very good at making mods like these love it keep it up! (also srry if this is not a issue)

thanks man
and no problem xd


"Images up to date" column missing

Bug description
The "images up to date" column is missing in my stack overview.
Expected behavior
It should be there.
Technical details:

Portainer version: Portainer Business 2.17.1
Docker version (managed by Portainer): Standalone 20.10.21
Platform (windows/linux): Linux
Browser: Firefox/Chromium/Safari
Use: Portainer at Home
Have you reviewed our technical documentation and knowledge base? Yes

This option can be toggled under Host > Setup on your environment, in the Other section at the bottom of the page. You can find more information in our documentation.

Yes, thank you very much. I never found that setting as i have the sidebar in minimalistic mode.


WSL stopped working - hangs for any distro access

Windows Version
aa
WSL Version
2.2.24
Are you using WSL 1 or WSL 2?

[ ] WSL 2
[ ] WSL 1

Kernel Version
No response
Distro Version
Ubuntu 22.04
Other Software
No response
Repro Steps
Type:

wsl

or
wsl -d whateverdistro
We have reinstalled Windows Subsystem for Linux & Virtual Machine Platform Feature. Still the same.
Expected Behavior
WSL works again, distro access restored.
Actual Behavior
WSL access Hangs completely.
We have 4 different distros:

Ubuntu-22.04
Ubuntu-20.04
data-distro (alpine)
wsl-vpnkit

trying access just freezes the terminal. Powershell consumes memory.
Also accesing the distros from \wsl$ behaves wrongly, initially seems to work but then shows

We have rebooted machine, same issue.
Diagnostic Logs
No logs, the Enterprise antivirus blocks the execution.

Solved:

The disk for one of the distros ran out of space (only 10Mb left).
One of the distros mounted this disk from the packed distro.
We ran wsl --update --pre-release & moved the disk to other with more space, registered back again & it works.


Day ahead optimization results all over the palce

I'm not sure if this is a bug or if I've managed to configure something incorrectly. I'm using the Docker version of Emhass and my day ahead optimization results sometimes work but mostly I end up with completely impossible values and a graph that is just flat with insane spikes everywhere. I've taken a screen grab of the table to show you what I mean:

I've tried different solvers all to no avail. One weird thing is the GLPK one doesn't do anything at all if I use it and I end up with the load and pv forecast showing fine, and the battery flat lined at the target value.
The logs don't give any info.

Post your configuration and any runtime parameters to your call. There's something that makes the optimization lost. Posting the emhass logs from the call may help.
Also, check the optim_status variable. I bet it says infeasible.

I managed to sort it out, I had modified certain parameters in the config yaml that the system didn't like. I'm slowly getting something workable.
I have never seen it say anything other than infeasible. I've tried tweaking the offpeak and peak times etc but it never changes anything. I also don't have peak or off peak tariffs where I live and to make it more interesting I have rolling blackouts that have a schedule I can pick up in HASS. Working out how to implement that would be quite handy. - suppose it would have to be treated almost like a peak time window but the battery would need to be charged in anticipation of the blackout window to a certain amount.
I also don't sell back to the grid because it's not worth the pittance I'd get and our utility provider insists we'd have to be net users anyway so I could sell maybe 50-100kwh and that's that.

I have never seen it say anything other than infeasible.

How is this possible, please share your configuration

I also don't have peak or off peak tariffs where I live and to make it more interesting I have rolling blackouts that have a schedule I can pick up in HASS. Working out how to implement that would be quite handy. - suppose it would have to be treated almost like a peak time window but the battery would need to be charged in anticipation of the blackout window to a certain amount.

Yes this can be treated by setting some extremely expensive grid energy prices during your scheduled blackout period. This way the system should converge to a solution where your battery was prepared to maximize self-consumption during that period. But solve your feasibility issue first.

I have never seen it say anything other than infeasible.

How is this possible, please share your configuration

Here it is. I wonder if it doesn't have something to do with the inverter selection? I chose the closest thing to what I actually have, which is a Sunsynk 8kW unit. There are no Deye/Sunsynk/Fusion inverters anywhere on the list you're using.
`# Configuration file for EMHASS
retrieve_hass_conf:
freq: 30 # The time step to resample retrieved data from hass in minutes
days_to_retrieve: 4 # We will retrieve data from now and up to days_to_retrieve days
var_PV: 'sensor.deyeinvertermaster_pv_power' # Photovoltaic produced power sensor in Watts
var_load: 'sensor.inverter_load_w_o_deferrable_load' # Household power consumption sensor in Watts (deferrable loads should be substracted)
load_negative: False # Set to True if the retrived load variable is negative by convention
set_zero_min: True # A special treatment for a minimum value saturation to zero. Values below zero are replaced by nans
var_replace_zero: # A list of retrived variables that we would want  to replace nans with zeros

'sensor.deyeinvertermaster_pv_power'
var_interp: # A list of retrived variables that we would want to interpolate nan values using linear interpolation
'sensor.deyeinvertermaster_pv_power'
'sensor.inverter_load_w_o_deferrable_load'
method_ts_round: 'nearest' # Set the method for timestamp rounding, options are: first, last and nearest

optim_conf:
set_use_battery: True # consider a battery storage
delta_forecast: 1 # days
num_def_loads: 1
P_deferrable_nom: # Watts

2000.0
def_total_hours: # hours
2
def_start_timestep: # timesteps
12
def_end_timestep: # timesteps
18
treat_def_as_semi_cont: # treat this variable as semi continuous
True
set_def_constant: # set as a constant fixed value variable with just one startup for each 24h
False
weather_forecast_method: 'scrapper' # options are 'scrapper', 'csv', 'list', 'solcast' and 'solar.forecast'
load_forecast_method: 'naive' # options are 'csv' to load a custom load forecast from a CSV file or 'naive' for a persistance model
load_cost_forecast_method: 'hp_hc_periods' # options are 'hp_hc_periods' for peak and non-peak hours contracts and 'csv' to load custom cost from CSV file
list_hp_periods: # list of different tariff periods (only needed if load_cost_forecast_method='hp_hc_periods')
period_hp_1:

start: '02:30'
end: '15:00'

period_hp_2:

start: '17:00'
end: '20:00'
load_cost_hp: 0.292 # peak hours load cost in €/kWh (only needed if load_cost_forecast_method='hp_hc_periods')
load_cost_hc: 0.141 # non-peak hours load cost in €/kWh (only needed if load_cost_forecast_method='hp_hc_periods')
prod_price_forecast_method: 'constant' # options are 'constant' for constant fixed value or 'csv' to load custom price forecast from a CSV file
prod_sell_price: 0.043 # power production selling price in €/kWh (only needed if prod_price_forecast_method='constant')
set_total_pv_sell: False # consider that all PV power is injected to the grid (self-consumption with total sell)
lp_solver: 'default' # set the name of the linear programming solver that will be used. Options are 'PULP_CBC_CMD', 'GLPK_CMD' and 'COIN_CMD'.
lp_solver_path: 'empty' # set the path to the LP solver, COIN_CMD default is /usr/bin/cbc
set_nocharge_from_grid: False # avoid battery charging from the grid
set_nodischarge_to_grid: True # avoid battery discharging to the grid
set_battery_dynamic: False # add a constraint to limit the dynamic of the battery power in power per time unit
battery_dynamic_max: 0.9 # maximum dynamic positive power variation in percentage of battery maximum power
battery_dynamic_min: -0.9 # minimum dynamic negative power variation in percentage of battery maximum power
weight_battery_discharge: 0.1 # weight applied in cost function to battery usage for discharge
weight_battery_charge: 0.1 # weight applied in cost function to battery usage for charge

plant_conf:
P_from_grid_max: 9000 # The maximum power that can be supplied by the utility grid in Watts
P_to_grid_max: 9000 # The maximum power that can be supplied to the utility grid in Watts
module_model: # The PV module model

'CSI_Solar_Co__Ltd__CS6W_545MB'
inverter_model: # The PV inverter model
'Ginlong_Technologies_Co___Ltd___RHI_1P8K_HVES_5G__240V_'
surface_tilt: # The tilt angle of your solar panels
15
surface_azimuth: # The azimuth angle of your PV installation
0
modules_per_string: # The number of modules per string
10
strings_per_inverter: # The number of used strings per inverter
1
Pd_max: 8000 # If your system has a battery (set_use_battery=True), the maximum discharge power in Watts
Pc_max: 4800 # If your system has a battery (set_use_battery=True), the maximum charge power in Watts
eta_disch: 0.95 # If your system has a battery (set_use_battery=True), the discharge efficiency
eta_ch: 0.95 # If your system has a battery (set_use_battery=True), the charge efficiency
Enom: 12000 # If your system has a battery (set_use_battery=True), the total capacity of the battery stack in Wh
SOCmin: 0.4 # If your system has a battery (set_use_battery=True), the minimun allowable battery state of charge
SOCmax: 1.0 # If your system has a battery (set_use_battery=True), the minimun allowable battery state of charge
SOCtarget: 0.6 # If your system has a battery (set_use_battery=True), the desired battery state of charge at the end of each optimization cycle
`

Try changing def_start_timestep and def_end_timestep to 0 for now.
Also try setting the default 0.0 for weight_battery_discharge and weight_battery_charge.
Your azimuth got to be wrong. That 0 means that your panels are facing north. The azimuth convention is defined as degrees east of north (e.g. North=0, South=180, East=90, West=270). Typically PV panels are installed facing south.
Also you only have 10 modules for a 8kW inverter? What is the power of your modules? Yes you can test different inverters from that list, the most important thing is just to find a similar nominal power as yours.
Please share your shell or REST command used to launch the optimization.

Thanks I'll give those settings a try.
The azimuth is spot on: I'm in the Southern Hemisphere (South Africa), and all my panels face true north. I actually have 12x 545W panels but because I get such bad shading here I chose 10 to get closer to my real maximum at this time of the year. I also have them set up as 2 strings of 6 but while I was troubleshooting the initial problems with the calculations I just set it to 1 string in the config. I will be pushing a PR soon to add the damping setting for the forecast.solar integration to try and deal with the early morning/late evening shading.
As for the weight_battery_discharge and weight_battery_charge I only changed those this morning to see if anything different would happen.
For my testing I click the button on the webpage but otherwise I've been launching the day-optim rest api from hass using this (as per your quickstart guide):
curl -i -H \"Content-Type:application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim

Oh yes sothern hemisphere, my bad.
Ok and so when you launch that with the logger set at INFO what does it says?

Here are the logs (I changed the weight back to 0.0 and set the timestep to 0 and 0.
024-05-14 10:18:01,553 - web_server - INFO - Launching the emhass webserver at: http://<IP_ADDRESS>:5000
2024-05-14 10:18:01,553 - web_server - INFO - Home Assistant data fetch will be performed using url: http://<IP_ADDRESS>:8123/
2024-05-14 10:18:01,553 - web_server - INFO - The data path is: /app/data
2024-05-14 10:18:01,556 - web_server - INFO - Using core emhass version: 0.9.0
waitress   INFO  Serving on http://<IP_ADDRESS>:5000
2024-05-14 10:18:08,647 - web_server - INFO - EMHASS server online, serving index.html...
2024-05-14 10:18:18,583 - web_server - INFO - Passed runtime parameters: {}
2024-05-14 10:18:18,584 - web_server - INFO -  >> Setting input data dict
2024-05-14 10:18:18,584 - web_server - INFO - Setting up needed data
2024-05-14 10:18:18,601 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2024-05-14 10:18:21,441 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-05-14 10:18:21,442 - web_server - INFO - Retrieve hass get data method initiated...
2024-05-14 10:18:25,681 - web_server - INFO -  >> Performing dayahead optimization...
2024-05-14 10:18:25,681 - web_server - INFO - Performing day-ahead forecast optimization
2024-05-14 10:18:25,682 - web_server - INFO - Perform optimization for the day-ahead
2024-05-14 10:18:25,704 - web_server - WARNING - Solver default unknown, using default
Welcome to the CBC MILP Solver
Version: 2.10.3
Build Date: Dec 15 2019
command line - /usr/local/lib/python3.11/dist-packages/pulp/solverdir/cbc/linux/64/cbc /tmp/0e9efe91d96d4575ac267673dbb89b5f-pulp.mps -max -timeMode elapsed -branch -printingOptions all -solution /tmp/0e9efe91d96d4575ac267673dbb89b5f-pulp.sol (default strategy 1)
At line 2 NAME          MODEL
At line 3 ROWS
At line 487 COLUMNS
At line 6392 RHS
At line 6875 BOUNDS
At line 7356 ENDATA
Problem MODEL has 482 rows, 384 columns and 5520 elements
Coin0008I MODEL read with 0 errors
Option for timeMode changed from cpu to elapsed
Problem is infeasible - 0.00 seconds
Option for printingOptions changed from normal to all
Total time (CPU seconds):       0.00   (Wallclock seconds):       0.01
2024-05-14 10:18:25,735 - web_server - INFO - Status: Infeasible
2024-05-14 10:18:25,735 - web_server - INFO - Total value of the Cost function = -0.08
2024-05-14 10:18:26,014 - web_server - INFO -  >> Sending rendered template table data

Ok I don't see anything wrong with your configuration to fall on that unfeasible condition.
Probably try with a different solver setting.
Set for example: lp_solver: 'COIN_CMD' and lp_solver_path: '/usr/bin/cbc'. Just to test.
If else then this is a problem with your data on sensors: sensor.deyeinvertermaster_pv_power and sensor.inverter_load_w_o_deferrable_load. What those looks like?
One quick test that we can do is to pass some dummy data to the optimizer that overrides those two sensors.
To do that you use this:
curl -i -H \"Content-Type:application/json\" -X POST -d '{\"pv_power_forecast\":[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 350, 706.1, 1230.9, 2567.5, 3766.35, 5249.45, 8989.65, 8486.5, 15394.65, 5821.65, 5233.40, 7795.5, 10456.30, 7783.8, 5833.65, 7583.15, 6955.65, 8600.65, 4103.75, 4022.04, 1258.15, 396.25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], \"load_power_forecast\":[3410.0, 382.0, 2099.0, 2790.0, 3931.0, 515.0, 1943.0, 1996.0, 4003.0, 2850.0, 1906.0, 2973.0, 1145.0, 533.0, 3737.0, 3548.0, 404.0, 3215.0, 3260.0, 2255.0, 3968.0, 2473.0, 1764.0, 3693.0, 1982.0, 1328.0, 1211.0, 109.0, 3882.0, 3909.0, 3528.0, 1632.0, 1074.0, 2803.0, 467.0, 992.0, 4222.0, 487.0, 2154.0, 862.0, 2924.0, 1732.0, 3834.0, 3795.0, 3778.0, 635.0, 2110.0, 2150.0]}' http://localhost:5000/action/dayahead-optim

Your command line overrides work and I see Optimal at the bottom. It must be my sensors. Perhaps the format/type of data they are capturing is not correct. Here is the response from the api call to HASS for the load:
[
    [
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "993",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:10+00:00",
            "last_reported": "2024-05-14T10:11:10+00:00",
            "last_updated": "2024-05-14T10:11:10+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "1002",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:20.399422+00:00",
            "last_reported": "2024-05-14T10:11:20.399422+00:00",
            "last_updated": "2024-05-14T10:11:20.399422+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "966",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:24.359031+00:00",
            "last_reported": "2024-05-14T10:11:24.359031+00:00",
            "last_updated": "2024-05-14T10:11:24.359031+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "1010",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:28.403567+00:00",
            "last_reported": "2024-05-14T10:11:28.403567+00:00",
            "last_updated": "2024-05-14T10:11:28.403567+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "1008",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:32.514132+00:00",
            "last_reported": "2024-05-14T10:11:32.514132+00:00",
            "last_updated": "2024-05-14T10:11:32.514132+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        },
        {
            "entity_id": "sensor.inverter_load_w_o_deferrable_load",
            "state": "977",
            "attributes": {
                "state_class": "measurement",
                "unit_of_measurement": "W",
                "device_class": "power",
                "friendly_name": "Inverter Load w/o Deferrable Load"
            },
            "last_changed": "2024-05-14T10:11:36.380832+00:00",
            "last_reported": "2024-05-14T10:11:36.380832+00:00",
            "last_updated": "2024-05-14T10:11:36.380832+00:00",
            "context": {
                "id": "01HXV285N701BVZJP660G9JKXA",
                "parent_id": null,
                "user_id": null
            }
        }
    ]
]

I don't see anything wrong there. But what do they look like directly in HA?

The data comes through every 2-3 seconds from my ESPHome sensor connected directly to the inverter. Does the sensor need to have fewer data points and log in Wh instead of W?

Okay I'm making some progress:
If I manually send in the load forecast using my data (I just used the values it gave me on the webpage) then it is still infeasible. If I take the PV data from the webpage and send it in manually then it shows as Optimal. I'm guessing the data coming from "scrapper" is not working too well for me. I'm going to try Solcast and Forecast.Solar and see what happens.

Confirmed. Using it the way I was and with Solcast and Forecast.Solar seems to be fine. It shows Optimal every time.

Ok. Then use those methods instead. But I'm still intrigued by why the scrapper method does not work for you.
Probably no data available on your lan/lon? That is a methods that have been working perfectly well since long time.

Ok. Then use those methods instead. But I'm still intrigued by why the scrapper method does not work for you.
Probably no data available on your lan/lon? That is a methods that have been working perfectly well since long time.

I'm also confused by it. But the scrapper data is there, and if you copy the values as they're displayed on the webpage under P_PV and use them via the command line then it works.
Like I mentioned in my earlier post I'll end up using forecast.solar and customize it slightly because it allows me to use a custom horizon so I can get more realistic forecasts based on the shadowing around me.

Ok no problem.
Please close this issue if the original problem is solved.
Feel free to open a new issue if needed


[CMake] Make SWIFT_HAVE_LIBXML an option.

Default this option to its previous value (whether libxml2 was found),
but allow us to independently turn it off.

@swift-ci please smoke test and merge


Update UIProgressView to use backing XAML controls

This adds a backing XAML control to, and completes the implementation for, UIProgressView.

:shipit:

:shipit:


MPT-30B Functions support

Hi,
I'm working on fine-tuning the MPT-30B for function calling. Currently still preparing the fine-tuning dataset.
AFAIK there is no open-source fine-tuned model for function support(let me know if you know).
I'm not sure about the format and wanted to get some advice. Basically proposing to add "function" and "tool" role to the original ChatML format. And putting the function schema to the system prompt. Here is the starting context. Functions are converted to a schema and have been put as a system prompt. (we can change this to something like "function_definitions" etc)
The model should start generating either "assistant" or "function" with after this exact context:
<|im_start|>system
// Plugin for managing a TODO list, you can add, remove and view your TODOs.
namespace todo {

// Add a todo to the list
type addTodo = (_: {
// The todo to add to the list.
todo: string,
}) => any;

} // namespace todo<|im_end|>
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Add X to my todo list<|im_end|>

If it starts with "function", it'll generate the namespace.function definition. Like this:
<|im_start|>function
todo.addTodo("X")<|im_end|>

Then we'll give this to the user. OR as a library, we can actually call the function too. The function response's role can be named as "tool" like this:
<|im_start|>tool
{"status": "OK"}
<|im_start|>assistant
Okay, I've added!.<|im_end|>

Full raw format:
<|im_start|>system
// Plugin for managing a TODO list, you can add, remove and view your TODOs.
namespace todo {

// Add a todo to the list
type addTodo = (_: {
// The todo to add to the list.
todo: string,
}) => any;

} // namespace todo<|im_end|>
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Add X to my todo list<|im_end|>
<|im_start|>function
todo.addTodo("X")<|im_end|>
<|im_start|>tool
{"status": "OK"}
<|im_start|>assistant
Okay, I've added!.<|im_end|>

What do you think?

Update:
instead of using seperate tool role, i think its better to go with function as the function response. And seperating function calls with different seperators. So we can use function calls in between messages.
<|im_start|>system
...FUNCTION_DEFINITION PART...<|im_end|>
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Add X to my todo list<|im_end|>
<|im_start|>assistant
Sure! Lets add it.
<|fn_start|>todo.addTodo
{"text": "X"}
<|fn_end|>
<|im_start|>function
{"status": "OK"}
<|im_start|>assistant
Okay, I've added!<|im_end|>

We are looking into tool use, thank you for sharing your work. I am curious, have you considered instead of
<|fn_start|>todo.addTodo
{"text": "X"}<|fn_end|>
<|im_start|>function
{"status": "OK"}<|im_end|>

My instinct is to have this be
<|im_start|>system function todo.addTodo
{"text": "X"}
response: {"status": "OK"}<|im_end|>

or something similar, since, won't the system block waiting for the response from the tool? So you can keep it in one message?
Also I am curious how you came up with the syntax for defining tools.

My initial thinking was having two EOS, both im_end and fn_end. So the sampling will stop there too. Let me know if you think one EOS is better for simplicity.
Also, since I can define the roles as the owner of the message, it would be much more intuitive to have assistant own the function call, rather than the system. In your revised version, after the function call request, there is only "\n" as far as I see. so that makes it not suitable for generation stop. But maybe we can put the im_end after the function call request in your revised version.
Syntax is essentially simplified TypeScript declaration file (.d.ts). I got the inspiration from chatgpt plugins.
The fact that the models trained on lots of "type" text data of typescript code makes this powerful enough to understand what is a "type" or signature of a function. Introducing some DSL that is completely different from the pretraining data would make it not understand well enough.
Also note that functions always takes one single object parameter that is typed. So the model will know that it should output an object and not strings/ints seperated by some arbitrary seperator when passing to the functions.
Here is an extended version. But we dont need the implementation part, so we can get rid of it:
// Plugin for managing a TODO list, you can add, remove and view your TODOs.
namespace todo {

    // Function to add a todo to the list
    function addTodo(_: { todo: string; }): any {
        // implementation here
    }

} // namespace todo

Have you considered adopting the same function-defining schema that OpenAI uses? Personally, I haven't encountered any issues with this schema and believe that maintaining consistency might help others, since they won't need to redefine a function that was initially used with OpenAI's function calling when they use their functions with the MPT-30b function calling. Just a thought.

I'd caution against adding tokens if you can avoid it. I recently finetuned MPT-30b on a reddit-style dataset, and initially made heavy use of special tokens as separators (<|post_title|>title here<|post_author|>username....) I found that even after training on 150M tokens it would still associate these with their correct meaning very loosely. Retraining with plaintext separators ([Title] title here [Author] username...) solved the issue completely.
Granted, I was going about it a bit excessively (10 or so custom tokens), but the poor results were surprising and lead me to believe that you'll be better off not adding <|fn_start|> and <|fn_end|> - especially if you don't have hundreds of millions of tokens to finetune on.

Have you considered adopting the same function-defining schema that OpenAI uses?

@unaidedelf8777 If you mean the JSON Schema Object definition, then that is something that you use on the API but not necessarily in the model. That JSON Schema definition can be converted to the schema that the model would use.
@float-trip That is a very useful information! What about special tokens that we would add to the dictionary? For example you would have added "<|post_title|>" as a seperate token that never existed in the pre-training step but it would exist in the fine-tuning step. So the model doesnt have any associations of "title", but it would know that its something different in the embedding space. same goes for <|fn_start|> and others. It could require a custom tokenizer other than the default mpt30b though.

If I'm understanding right, that's what I did - I modified the tokenizer like this:
from transformers import GPTNeoXTokenizerFast

tokenizer = GPTNeoXTokenizerFast.from_pretrained(
    "EleutherAI/gpt-neox-20b",
    additional_special_tokens=[
        "<|post_title|>",
        "<|post_url|>",
        "<|post_author|>",
        # ...
    ],
)

tokenizer.save_pretrained("tokenizer")

The model was somewhat able to learn the correct meanings for these (especially for the more frequently occurring tokens like<|comment_author|>, <|comment_body|>, etc.) It's possible that only adding a couple new tokens would be fine, and the problem only appears when going overboard.

Thanks @float-trip . Got it. Then we can remove the fn_start and instead used im_start with assistant role. (Like OpenAI API returns assistant role on function calls)
Then we can use the existing tokens that we have used but using the fields for those roles. Basically an arbitrary map for every role that they just examples on chatml docs.
So the new revised version should be very similar to the openai's model. Except that I think they don't use im_start anymore. They reduced the token count for gpt3 0613 models. and all models for gpt4. you can see here
<|im_start|>system
...FUNCTION_DEFINITION PART...<|im_end|>
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Add X to my todo list<|im_end|>
<|im_start|>assistant
Sure! Lets add.
<|im_start|>assistant to=todo.addTodo
{"text": "X"}<|im_end|>
<|im_start|>function name=todo.addTodo
{"status": "OK"}
<|im_start|>assistant
Okay, I've added!<|im_end|>

@musabgultekin, Yeah I don’t know anything about the tokenizers really. I just finished building out a massive system with the openai schemas, ( which it dynamically constructs ). I just didn’t want to rebuild that logic bc it’s quite complex, and I figured most others wouldn’t want to do similar things when migrating.

@musabgultekin @samhavens @float-trip @unaidedelf8777
You should all check out this twitter thread. It forces the LLM to output valid "code" following function specs. It might be worth looking into.
https://twitter.com/GrantSlatton/status/1657559506069463040

If the goal is to run parsable outputs for the given instruct, it solves the issue and a good solution since it doesn't require further fine-tuning. Really Like it.
This would work for function only models. For example if you have a for loop that decides what to do always, then this works. You can have a for loop for a robot, that it can only do "go_forward", "go_backwards" "wait" etc, then its good. But if it needs to decide not to call any functions but instead needs to ask followup questions, then it wont work. Fine-tuning is the way to go in that case.
Thanks for the info @AlbertMarashi ! Checking it out

@AlbertMarashi For a more general/flexible version of that idea, check out LogitsWarper and LogitsProcessor in HuggingFace's transformers library.

I've finally prepared a good looking dataset ready for fine-tuning. It has 5300~ examples. 500~ different schemas.
All different prompts, %50 contains function calls, %50 has function schemas but no callsto teach the model when not to call the functions.
But 8xA100 40GB gave OOM. I've looked two clouds for 8xA100 80GB, but no availability. I'm gonna have to defer training until I find some big GPUs for now.

I have a bash script in here which provisions an 8xA100 80gb from LambdaLabs: https://gist.github.com/float-trip/679019a23f246b17d2dff9e2cf55c387
It generally still takes a few hours, but if you leave it running you'll get it eventually.

Oh amazing! @float-trip Will use it thanks 🙏

I’m sleeking on a dataset of function definitions, prompts to call the functions, example responses from the functions, and a message that the model gives based on the response from the function.
All I did was scrape a ton of openapi schemas from the apisguru repository, and then turned them into function definitions, then devised a gpt prompt to make up prompts to call those functions, example responses from the functions, and the model responses based on those function responses.
It is chewing away rn. Will ping y’all when I throw it on hugging face… more to come.

I've finally prepared a good looking dataset ready for fine-tuning. It has 5300~ examples. 500~ different schemas. All different prompts, %50 contains function calls, %50 has function schemas but no callsto teach the model when not to call the functions.
But 8xA100 40GB gave OOM. I've looked two clouds for 8xA100 80GB, but no availability. I'm gonna have to defer training until I find some big GPUs for now.

You should try RunPod.Io, their pricing is slightly higher than lambdas, but they usually have availability for basically everything. It says here that for a A100 SXM they only want $1.44 USD per card hour. Not too bad in my opinion.
One other thing, if you haven’t used gcp yet, they’ll give you like 300 dollars of credits for a free trial. You will probably have to get in touch with their team if you want more than one A100 though. But in my experience they’re pretty good about fast replies.

I’m sleeking on a dataset of function definitions, prompts to call the functions, example responses from the functions, and a message that the model gives based on the response from the function.
All I did was scrape a ton of openapi schemas from the apisguru repository, and then turned them into function definitions, then devised a gpt prompt to make up prompts to call those functions, example responses from the functions, and the model responses based on those function responses.
It is chewing away rn. Will ping y’all when I throw it on hugging face… more to come.

It will have around 27000 examples of functions and completions. No extra training to tell the model when to call the functions really, only how to.

@unaidedelf8777 Thats great! I'm wondering if you only used one function per example and thats how you reached to 27k. Cause the site has 2.5k APIs. Also, did you try making the requests? cause it would be bad if most of them are auth required.

If I'm understanding right, that's what I did - I modified the tokenizer like this:
from transformers import GPTNeoXTokenizerFast

tokenizer = GPTNeoXTokenizerFast.from_pretrained(
    "EleutherAI/gpt-neox-20b",
    additional_special_tokens=[
        "<|post_title|>",
        "<|post_url|>",
        "<|post_author|>",
        # ...
    ],
)

tokenizer.save_pretrained("tokenizer")

The model was somewhat able to learn the correct meanings for these (especially for the more frequently occurring tokens like<|comment_author|>, <|comment_body|>, etc.) It's possible that only adding a couple new tokens would be fine, and the problem only appears when going overboard. But I'd at least do a test run without any new tokenization to see if it performs better.

@float-trip I think you also needed to resize the embeddings to
https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L866C25-L866C25

@unaidedelf8777 Thats great! I'm wondering if you only used one function per example and thats how you reached to 27k. Cause the site has 2.5k APIs. Also, did you try making the requests? cause it would be bad if most of them are auth required.

No, the function responses are synthetic, but were verified to make sure they have the correct schema. I have the dataset repo setup rn, it has the prompt I used in there. the repo is empty except for a readme and the prompt because the API is slow asl and I didn't feel like making it asynchronous. and to answer your question, yes currently it is only one prompt one function call, however I plan on iterating and improving the dataset soon enough.
IF this dataset picks up any traction, do you think that I should make a petreon, since openai is expensive for this kinda thing.
also here's the hf link

@unaidedelf8777 Thats great! I'm wondering if you only used one function per example and thats how you reached to 27k. Cause the site has 2.5k APIs. Also, did you try making the requests? cause it would be bad if most of them are auth required.

No, the function responses are synthetic, but were verified to make sure they have the correct schema. I have the dataset repo setup rn, it has the prompt I used in there. the repo is empty except for a readme and the prompt because the API is slow asl and I didn't feel like making it asynchronous. and to answer your question, yes currently it is only one prompt one function call, however I plan on iterating and improving the dataset soon enough.
IF this dataset picks up any traction, do you think that I should make a petreon, since openai is expensive for this kinda thing.
also here's the hf link

ill ping yall when its finished.

@musabgultekin @AlbertMarashi @float-trip @samhavens
Just uploaded the part of the dataset which is finished, its only around 1k examples though. more to come.
https://huggingface.co/datasets/unaidedelf87777/openapi-function-invocation-25k/
the preview is also messed up so it shows the prompt I used instead of the csv. if someone knows how to fix that please lmk.

@float-trip I think you also needed to resize the embeddings too
Checkout : https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py#L65C42-L65C42

Thanks, this is true in cases where model.config.vocab_size < num_tokens. GPT-NeoX-20B and the MPT models leave some extra room in the vocab size for performance reasons.

Hi @unaidedelf8777 , Thank you for the dataset!
If possible, can you save your dataset in jsonlines format? (seperate json on each line). Cause apparently the CSV has problems with your data/saving method and its not valid. Currently it doesnt look like its usable unfortunately :(

Hi @unaidedelf8777 , Thank you for the dataset!
If possible, can you save your dataset in jsonlines format? (seperate json on each line). Cause apparently the CSV has problems with your data/saving method and its not valid. Currently it doesnt look like its usable unfortunately :(

Yeah, sorry ab that. It’s because some entries span a couple lines. I have a cleaned version, I just am yet to throw it up there yet. If you look in the Data folder of the repo, there are train and Validation jsonl files, pre formatted with the special tokens which I described in the dataset card.

Hi @unaidedelf8777 , Thank you for the dataset!
If possible, can you save your dataset in jsonlines format? (seperate json on each line). Cause apparently the CSV has problems with your data/saving method and its not valid. Currently it doesnt look like its usable unfortunately :(

Yeah, sorry ab that. It’s because some entries span a couple lines. I have a cleaned version, I just am yet to throw it up there yet. If you look in the Data folder of the repo, there are train and Validation jsonl files, pre formatted with the special tokens which I described in the dataset card.

I’m working on a loading script for the dataset right now which will serve the jsonl files. I just don’t know how to write it, and gpt seems not to know either so I’ll have to actually look into it ☹

I've wrote a new training system that calculates loss on assistant responses and function calls from assistant.
Used LLaMA 7B as base and Huggingface trainer. Trained it on 37k examples (34k ShareGPT conversations(wizard vicuna uncensored) + 3k GPT4 generated prompts and function calls). Sadly I wasn't able to find 8x A100 80GB consistently for the training experiments. It was stressful, cause I've already had the MPT training dataset, code and I simply wasn't able to find some compute to train it. So thats why I decided to start with a smaller model. (MPT 7B also failed on nodes that I have tried)
Here is the repo and inference code that works rn:  https://github.com/musabgultekin/functionary/blob/main/inference.ipynb
My preliminary manual tests shows that its working on instructions. It knows when to call a function and which function. Will prepare the codebase. Need to write evaluation suite and custom tooling around it though.
There are issues of course; like multiple rounds not working properly due to the dataset is only has one round per conversation for function calls. Also its hallucinating when commentating over function outputs for some reason, e.g.: adding stuff that doesn't exist on the function outputs. I hope we can reduce it by training on 13B, more sharegpt convs and also potentially using @unaidedelf8777 's dataset.
Per @float-trip 's suggestion, I've not introduced new tokens (will need eval suite first, then will do the ablation study).
I'll share more details, design choices, and code soon. I feel like current status is just proof of concept. Please let me know
what are your suggestions.
Here is how it decides to call functions:

Here is how it uses the function output:

I've wrote a new training system that calculates loss on assistant responses and function calls from assistant. Used LLaMA 7B as base and Huggingface trainer. Trained it on 37k examples (34k ShareGPT conversations(wizard vicuna uncensored) + 3k GPT4 generated prompts and function calls). Sadly I wasn't able to find 8x A100 80GB consistently for the training experiments. It was stressful, cause I've already had the MPT training dataset, code and I simply wasn't able to find some compute to train it. So thats why I decided to start with a smaller model. (MPT 7B also failed on nodes that I have tried)
Here is the repo and inference code that works rn: https://github.com/musabgultekin/functionary/blob/main/inference.ipynb
My preliminary manual tests shows that its working on instructions. It knows when to call a function and which function. Will prepare the codebase. Need to write evaluation suite and custom tooling around it though.
There are issues of course; like multiple rounds not working properly due to the dataset is only has one round per conversation for function calls. Also its hallucinating when commentating over function outputs for some reason, e.g.: adding stuff that doesn't exist on the function outputs. I hope we can reduce it by training on 13B, more sharegpt convs and also potentially using @unaidedelf8777 's dataset.
Per @float-trip 's suggestion, I've not introduced new tokens (will need eval suite first, then will do the ablation study).
I'll share more details, design choices, and code soon. I feel like current status is just proof of concept. Please let me know what are your suggestions.
Here is how it decides to call functions:
Here is how it uses the function output:

looks good!

Its been nearly a month since I started, but I managed to make function calling working properly with LLaMA. Unfortunately not MPT because of OOMs that I got constantly.
Here is the full repo for inference and details of dataset: https://github.com/musabgultekin/functionary
I'm gonna add training code and more info soon.
@unaidedelf8777 I saw your mpt-7b-CodeCaller-v1 . How is it going? Cause first, I trained a single turn conversation but it didnt work out well. So I had to use multiple turns in one conversation to make it actually work properly.

Its been nearly a month since I started, but I managed to make function calling working properly with LLaMA. Unfortunately not MPT because of OOMs that I got constantly.
Here is the full repo for inference and details of dataset: https://github.com/musabgultekin/functionary
I'm gonna add training code and more info soon.
@unaidedelf8777 I saw your mpt-7b-CodeCaller-v1 . How is it going? Cause first, I trained a single turn conversation but it didnt work out well. So I had to use multiple turns in one conversation to make it actually work properly.

@musabgultekin , that repo was just a failed LoRA fine-tune that I whipped out yesterday. I am still ironing out my dataset, and trying to really get rid of the useless entries. Right now My current plan is to add a few new attention heads on the upper layers, since I have noticed that the model really tries to call the functions/ doesn't know the format so it just gives mumbled garbage which it makes up.  all I need to do is find on what layer/chunk of layers it is generating the garbage, and replace the garbage it makes up with a function call, directed by the attention head(s).
Also, did you get it working on Llama 1 or 2? just curious. love the functionary screaming llama on your repo also!

@unaidedelf8777
I' think its probably the model doesnt know the schema of fn_def section that you put in the dataset. Thats why I used TypeScript definitions, which exists in the pretraining dataset of these LLMs. Also Microsoft TypeChat uses the same TypeScript definitions idea: https://github.com/microsoft/TypeChat/blob/d2f2de9ca37ef9adeb108d5fc60703b72fec0a22/site/src/blog/introducing-typechat.md#just-add-types
So I don't think adding new attention heads would be sufficient.
If you are convinced, I just committed a function for you that would help you on conversion from apisguru specs to typescript schemas. https://github.com/musabgultekin/functionary/commit/6cde13ca40be1ca4e873955c6d15e8969a578c50
Of yourse you can modify the schema as you need.
Its based on llama 1 not 2. I'm currently checking out llama 2 training, doing some experiments etc.
Thanks for the petals project info! i thought its only for forward pass. Thats great to know, will check out.

@unaidedelf8777 I' think its probably the model doesnt know the schema of fn_def section that you put in the dataset. Thats why I used TypeScript definitions, which exists in the pretraining dataset of these LLMs. Also Microsoft TypeChat uses the same TypeScript definitions idea: https://github.com/microsoft/TypeChat/blob/d2f2de9ca37ef9adeb108d5fc60703b72fec0a22/site/src/blog/introducing-typechat.md#just-add-types So I don't think adding new attention heads would be sufficient.
If you are convinced, I just committed a function for you that would help you on conversion from apisguru specs to typescript schemas. musabgultekin/functionary@6cde13c Of yourse you can modify the schema as you need.
Its based on llama 1 not 2. I'm currently checking out llama 2 training, doing some experiments etc.
Thanks for the petals project info! i thought its only for forward pass. Thats great to know, will check out.

Looking at the training mixture of llama a little closer, I do see what your saying, and the typescript definitions definitely do make a lot more sense when looking at them with no understanding of json. only reason I am really pushing with json definitions is because it is simple.. and more external systems which the model may interact with use it since it is more serializable. but I also know near nothing about typescript, so maybe there is some way to serialize json responses from servers into typescript, idk.

typescript definitions would be a lot more powerful I think

@musabgultekin , just gave you your first pr on the repo.
https://github.com/musabgultekin/functionary/pull/1
I would repeat what I said in the request, but my hand hurts from typing lo

Thank you all for the interesting discussion! I'm going to go ahead and close this issue as there is no question/feature request here.


Error 404/500 template not overridden on Alpha

Issue Summary
The template error.hbs for error page 404/500 on the theme is not overriding the core/server/views/error.hbs on latest alpha release
Steps to Reproduce

Stop server (if running)
Create template error.hbs on active theme
Start the server
Type an unknown route

Behaviour
Expected: Custom error page template to be rendered
Actual: Default error page template is rendered
Technical details:

Ghost Version: v1.0-alpha-21
Node Version: 7.2.0
Browser/OS: macOS
Database: sqLite

Hey @gomesphoto! Sorry to hear that you're having trouble.
Unfortunately, I am not able to reproduce this. Switched to Node 7.2 to see if this might be it, but it works fine for me. As soon as I add an error.hbs file in to my themes directory and restart Ghost, that's the one which is used.
I will close this issue for now, but we can reopen it, when it turns out to be an issue in the end.
We were talking on Slack already. Might be worth it to continue there again and you'll share some more information, like screenshots or share your theme to investigate?


[crunchyroll] Fix video_title (closes #25047)

Before submitting a pull request make sure you have:

[X] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[X] Searched the bugtracker for similar pull requests
[X] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[X] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[X] Bug fix
[ ] Improvement
[ ] New extractor
[ ] New feature

Description of your pull request and other information
The title has been changed so I made the following adjustments to the code:

Get video_title from meta tag instead, avoids inflexible regex
Add fallback to get video_title from <title> this is more flexible than the previous regex
Changed title format in tests to use ,  instead of  – as this appears multiple time in the meta and title tags
Removed test 0 and 1 as the media is no longer available
Added new test
Changed one of the test uploader to Re:Zero Partners from TV TOKYO as this is changed on the site as well

Passes all the single Crunchyroll tests and looks good on flake8. If there is anything else that needs to be done please let me know.
Closes #25047

I have made changes to the regex so it works on other locales. I have also added a test in pt-PT.
If you would like me to squash the commits, do let me know.

 C:\Users\Chidr>"C:\Users\Chidr\AppData\Local\Programs\Python\Python36\Scripts\youtube-dl.exe" --cookies E:\Download\cookies.txt --verbose -f "[height <=? 360][tbr>500]" --sub-lang ruRU --write-sub --sub-format "ass" "https://www.crunchyroll.com/a-certain-scientific-railgun/episode-13-system-those-who-arrive-at-heavens-will-in-an-only-human-body-792619"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--cookies', 'E:\\Download\\cookies.txt', '--verbose', '-f', '[height <=? 360][tbr>500]', '--sub-lang', 'ruRU', '--write-sub', '--sub-format', 'ass', 'https://www.crunchyroll.com/a-certain-scientific-railgun/episode-13-system-those-who-arrive-at-heavens-will-in-an-only-human-body-792619']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: none
[debug] Proxy map: {}
[crunchyroll] 792619: Downloading webpage
Traceback (most recent call last):
  File "__main__.py", line 19, in <module>
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\__init__.py", line 474, in main
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\__init__.py", line 464, in _real_main
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 2019, in download
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 797, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\crunchyroll.py", line 419, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\crunchyroll.py", line 270, in _download_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 794, in _download_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 660, in _download_webpage_handle
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 627, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 2238, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 464, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 482, in _open
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\utils.py", line 2572, in http_open
  File "C:\Python\Python34\lib\urllib\request.py", line 1183, in do_open
  File "C:\Python\Python34\lib\http\client.py", line 1137, in request
  File "C:\Python\Python34\lib\http\client.py", line 1177, in _send_request
  File "C:\Python\Python34\lib\http\client.py", line 1109, in putheader
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 2276-2280: ordinal not in range(256)

C:\Users\Chidr>"C:\Users\Chidr\AppData\Local\Programs\Python\Python36\Scripts\youtube-dl.exe" -u XXX -p XXX --verbose -f "[height <=? 360][tbr>500]" --sub-lang ruRU --write-sub --sub-format "ass" "https://www.crunchyroll.com/a-certain-scientific-railgun/episode-13-system-those-who-arrive-at-heavens-will-in-an-only-human-body-792619"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', '--verbose', '-f', '[height <=? 360][tbr>500]', '--sub-lang', 'ruRU', '--write-sub', '--sub-format', 'ass', 'https://www.crunchyroll.com/a-certain-scientific-railgun/episode-13-system-those-who-arrive-at-heavens-will-in-an-only-human-body-792619']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.18362
[debug] exe versions: none
[debug] Proxy map: {}
[crunchyroll] Downloading login page
[crunchyroll] Logging in
[crunchyroll] 792619: Downloading webpage
ERROR: Unable to extract video_title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\YoutubeDL.py", line 797, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\crunchyroll.py", line 446, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 1014, in _html_search_regex
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpjwbwqymm\build\youtube_dl\extractor\common.py", line 1005, in _search_regex
youtube_dl.utils.RegexNotFoundError: Unable to extract video_title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Where is the mistake?

@Merzav Hello, I just gave both of these methods a go and they both work on my machine. Are you sure you are using the fix on my branch that I've submitted for a PR?

I built your pr on my machine it works perfectly here.

Yes that was my fault, everything is working.