Model performs differently for model.fit and custom training loop

System information

Os platform Linux Ubuntu 16.04
TensorFlow version used - 2.2.0
Python version: 3.6.9

Custom Training Loop vs Model.fit code
model=model()
optimizer=tf.keras.optimizers.Adam()
batch_size=opt.b_size
n_batches = int(len(train_set) / opt.b_size)

for i in range(opt.epochs):
    loss_t=0
    loss_vt=0
    it=0
    for j in range(n_batches):
        with tf.GradientTape() as tape:
            tape.watch(model.trainable_variables)
            curr=train_set[it:it+batch_size]
            forward=model(curr,True)
            loss=tf.keras.losses.MeanAbsoluteError()(y_train[it:it+batch_size],forward)
            loss_t += loss

        grads=tape.gradient(loss,model.trainable_variables)
        optimizer.apply_gradients(zip(grads,model.trainable_variables))
        it+=batch_size

    # shuffling the test set
    index = np.arange(0, len(test_set))
    np.random.shuffle(index)
    test_set = test_set[index]
    y_test = y_test[index]
    loss_t=loss_t.numpy()

    # calculating the validation loss
    forward_v = model(test_set[:batch_size], False)
    loss_v = tf.keras.losses.MeanAbsoluteError()(y_test[:batch_size], forward_v)
    loss_v=loss_v.numpy()
    loss_t /= n_batches
    print("Loss: {} Validation loss: {} ".format( round(loss_t,4) , round(loss_v,4) ) )

The above mentioned code is the custom training loop of a model.
model=model()
model.compile(tf.keras.optimizers.Adam(),tf.keras.losses.MeanAbsoluteError(),metrics=['accuracy'])
model.fit(train_set,y_train,batch_size=opt.b_size,epochs=opt.epochs,validation_data=(test_set,y_test))

The above code uses model.fit method for training.
Behaviour and code  to replicate the Results
But when I run the same code on the same train dataset and validation dataset with all the same parameters, the validation loss obtained is very different in the two cases.

Here is the google colab gist to replicate the results.
Alternatively, to replicate results locally, please run  train.py and train2.py on my github using the following code after cloning the repository to reproduce the results:

bash import_weights.sh
python train.py --n_samples 1000 --epochs 100  --b_size 50 for model.fit
python train2.py --n_samples 1000 --epochs 100  --b_size 50 for custom training

added commands to replicate the bug from my github.

@arshagarwal
Can you please provide the issue faced in a colab gist for us to analyse the issue reported.

@Saduf2019
Thanks for taking the time, here is the google colab gist

@arshagarwal
Can you please provide the issue faced in a colab gist for us to analyse the issue reported.

Thanks for taking the time, here is the google colab gist.

@arshagarwal
I ran the code shared,please find the gist here.
Please confirm if this replicates the issue.

@arshagarwal
I ran the code shared,please find the gist here.
Please confirm if this replicates the issue.

Yes, this is exactly what I am trying to report.

Was able to replicate the issue in TF 2.6.0-dev20210530,please find the gist here..Thanks !

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


Provide public feed for Pilets

New Feature Proposal
Description
A public feed for Pilets should be available, so that developers who are starting to evaluate and use Piral, have a possibility to host their Pilets for development and testing purposes.
Background
The standard approach for Piral is that Pilets are loaded from a so-called feed service. When starting with Piral such a feed service is usually not available. To ease the process of evaluating Piral and get started with the first Pilets, a publicly accessible feed service should be available, where developers can perform a simple registration and publish their Pilets into their own dedicated feed.
Discussion
Some restrictions to the number of Pilets or number of Requests should apply, to keep the efforts of hosting and operating the public feed in a certain limit.

Should be there with the version 0.7.


Forecasting page in User's guide

The Forecasting page has at least two issues:

the "View on Github" link is broken
the example code refers to a pvlib location instance that is not defined. It is referenced in a block of code that isn't run, so doesn't generate an error. Including this code can be confusing to new users.

Looks like the NDFD section has some broken code too.

the "View on Github" link is broken

In fact most of the top-level pages in stable have broken "view on github" links.  This is because the 0.9.0 docs still point to the old repository locations from before #1173 moved those pages to the new user_guide subdirectory.  I don't think there's anything to do about it until we release 0.9.1 and the docs get rebuilt with the new repository locations.

On second look, the block of code referencing the location instance isn't execute, it's there to show how ghi is obtained from cloud cover. Maybe we can display that information differently to avoid confusing new users, who copy the code and try to run it.

@cwhanse  Can I work on this??

@soma2000-lang certainly. My idea is to replace these lines with math (for the last three lines) and provide a reference to the cloud_cover_to_irradiance_clearsky_scaling function
E.g.,
GHI = [offset + (1 - offset) x (1 - cc) ] x GHI_{clearsky}
where offset (% of clearsky GHI) is the minimum GHI under complete cloud cover, and cc is the cloud cover (fraction of sky dome).
DNI is computed using the DISC model , and DHI is computing as

.. code-block:: python
    solpos = location.get_solarposition(cloud_cover.index)
    cs = location.get_clearsky(cloud_cover.index, model='ineichen')
    # offset and cloud cover in decimal units here
    # larson et. al. use offset = 0.35
    ghi = (offset + (1 - offset) * (1 - cloud_cover)) * ghi_clear
    dni = disc(ghi, solpos['zenith'], cloud_cover.index)['dni']
    dhi = ghi - dni * np.cos(np.radians(solpos['zenith']))


http filter: ALPN http filter to override ALPN for upstream connection

Signed-off-by: crazyxy<EMAIL_ADDRESS>Description: Add ALPN http filter to override ALPN for upstream connection
Risk Level: Low
Testing: Unit test
Docs Changes: https://github.com/envoyproxy/envoy/compare/master...yxue:alpn?expand=1#diff-34da83bdc37c06f88007d918bfb6890d
Release Notes: https://github.com/envoyproxy/envoy/compare/master...yxue:alpn?expand=1#diff-d8d3e33358b55e6c0466dd1bb5f40c79
Fixes #Issue: Fix https://github.com/envoyproxy/envoy/issues/8197

at high level I doubt anyone outside istio will use this, thoughts @envoyproxy/maintainers?

+1, let's skip merging this to upstream until someone else asks for this.


Patch rules_android_ndk to make it compatible with '--incompatible_disallow_empty_glob'

Description
There was an attempt to make rules_android_ndk compatible with --incompatible_disallow_empty_glob, but it seems that it was not complete.

https://github.com/bazelbuild/rules_android_ndk/pull/37

Let's work around this in the Mozc side for now.
There must be no behavior change in the final artifacts.
Issue IDs

https://github.com/google/mozc/issues/1150

Steps to test new behaviors (if any)
A clear and concise description about how to verify new behaviors (if any).

OS: Ubuntu 24.04.1
Steps:

bazel build --config oss_linux package --config release_build --incompatible_disallow_empty_glob package
Confirm the build succeeds.

We have merged your PR.
Thank you for the contributions!


v8.0.3 or v8.0.2 on iOS 13.2.3 or 12.1.4 range selection totally broken with no error message.

(Required) Version Number:
v8.0.3 v8.0.2
Description
After upgrade to 8.0.2 and 8.0.3 to support swift 5.1 on iOS Device 13.2.3 & 12.1.4 will totally not able to range select.
Even though download Official Sample Project which also has same issue.
Steps To Reproduce
Install official sample, go to range selection demo.
Expected Behavior
Expect can success to select the range date.
Additional Context

Sorry for the late reply, but i was caught up.
You are using verion 8, but you didnt change the funciton signatures.
Here they are -> https://patchthecode.com/jtapplecalendar-version-8-0-0-migration-guide/
Please change all the function names.

Sorry for the late reply, but i was caught up.
You are using verion 8, but you didnt change the funciton signatures.
Here they are -> https://patchthecode.com/jtapplecalendar-version-8-0-0-migration-guide/
Please change all the function names.

I got the same issue, already changed all function signatures but the range selection is still broken.
I even tried renaming your outdated demo (https://github.com/patchthecode/JTAppleCalendar/blob/master/SampleJTAppleCalendar/Example Calendars/TestRangeSelectionViewController.swift) but it doesn't work too.
I'm currently on iOS 13.3 with JTAC 8.0.3 installed.


PKCalc [R]

App name
dashr-pk-calc
App

[ ] This app is currently deployed on Dash(R) Gallery: [deployment URL]
[x] This app is new

Source code repository
[source code repo URL]
Current appearance
[attach screenshot of current app here]
Redesign

Playground deployment
[url of playground deployment]

pharmacokinetics analysis.ai.zip


Get projects test

Background
Link to issue
https://biomage.atlassian.net/browse/BIOMAGE-991
Link to staging deployment URL
N.A.
Links to any Pull Requests related to this
N.A.
Anything else the reviewers should know about the changes here

Write unit test for route GET /projects
Write unit test for getProjects and getProjectsFromIds

Changes
Code changes
Definition of DONE
Your changes will be ready for merging after each of the steps below have been completed:
Testing

[X] Unit tests written
[ ] Tested locally with Inframock (with latest production data downloaded with biomage experiment pull)
[ ] Deployed to staging

To set up easy local testing with inframock, follow the instructions here: https://github.com/biomage-ltd/inframock
To deploy to the staging environment, follow the instructions here: https://github.com/biomage-ltd/biomage-utils
Documentation updates
Is all relevant documentation updated to reflect the proposed changes in this PR?

[ ] Relevant Github READMEs updated
[ ] Relevant wiki pages created/updated

Approvers

[ ] Approved by a member of the core engineering team
[ ] (UX changes) Approved by vickymorrison (this is her username, tag her if you need approval)

Just before merging:

[ ] After the PR is approved, the unstage script in here: https://github.com/biomage-ltd/biomage-utils is executed. This script cleans up your deployment to staging

Optional

[ ] Photo of a cute animal attached to this PR

I have a question and a suggestion. I am trying my first review from VisualStudio. I hope that I do not break anything :-)


[Bug Report] NPT-API errors on 1.17

Describe the bug
NBT-API errors with any voucher command use on 1.17.
Versions (please complete the following information):

Minecraft Server Version: 1.17
Plugin Version: v1.8.19

Pastebin any errors in console
https://paste.gg/p/anonymous/57918e305a594600b8ff041c34c4a42a

You need to join our discord and verify that you own the plugin then you will have access to the beta version of the plugin which I've updated to work on 1.17.
https://discord.gg/Wj2cEHUsqP

Fixed.


Skip AppVeyor builds modifying only the README

Commits only modifying the README.md shouldn't be built. AppVeyor supports such exclusion rules! Rephrases a passage in the README.md to test this.
Travis unfortunately does not support this.

Codecov Report

Merging #192 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #192   +/-   ##
=========================================
  Coverage     82.05%   82.05%
  Complexity      317      317
=========================================
  Files            39       39
  Lines           741      741
  Branches        155      155
=========================================
  Hits            608      608
  Misses           61       61
  Partials         72       72

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd51946...d0fa5c0. Read the comment docs.


debian vm stuck at grub> consol

Following the readme I'm unable to start a debian install:
iohyve set debguest loader=grub-bhyve
iohyve set debguest os=debian
iohyve install debguest debian-8.2.0-amd64-i386-netinst.iso
iohyve console debguest

Results with:
  Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists
   possible device or file completions.

grub>
grub>

The grub menu is unresponsive

I can't reproduce this with my copy of debian-8.2.0-amd64-i386-netinst.iso
If you are still having problems, can you tell me the output of iohyve getall debguest and iohyve version?

Getting debguest props...
size         20G
cpu          1
ram          256M
description  Sun_Jan_10_00:49:19_IST_2016
bargs        -A_-H_-P
boot         0
os           debian
name         debguest
install      yes
autogrub     \n
loader       grub-bhyve
con          nmdm0
persist      1
tap          tap0```

```bash
iohyve v0.7 2015/10/27 Im Here for the Party Edition

Try updating to 0.7.3 either from GitHub or from the latest port snapshot.
I'm still unable to reproduce with the settings you provided.

Also, after updating, run iohyve destroy debguest to remove it from VMM before going further.

Ok, iv found the issue
I didn't use fetch in order to get the iso (iv just downloaded), following the freebsd section I saw that the tool does it magic while setting up the iso
Thanks

Did you use iohyve cpiso to copy the ISO to iohyve's ISO storage?

No, I assumed that if I create the folder under /iohyve/ISO it will be ok, once I ran fetch for the BSD guest I notice that the tool create another parent folder (I guess with some ZFS properties)
The README section that deals with Linux doesn't mention the fetch command this was why I was confused
Thanks

Thanks for the info! I should be a bit more clear in the readme on those functions (fetch and cpiso).

Cool, right now I can't seem to get DHCP working on any of the VMs (both Freebsd and Debian) any idea what to look for? (iv set the bridge as instructed in the readme)
(My I suggest that you create a google group? that way I won't pollute the issues history)
Thanks

Feel free to open a separate issue. I don't think the chatter is loud enough here yet to justify a Google Group. This way, everything is in one place for issues and general discourse


Video hosting - Something other than YouTube?

Is there any way to host video content on something other than YouTube? Many corporate policies block YouTube. There are some other videos hosted on Microsoft's own services that aren't blocked so it would be nice to have an alternative.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 83582e61-22b0-fb99-43c7-65a6694affef
Version Independent ID: 61cc2546-2074-08b2-a6ed-e73da4825bfd
Content: Overview of Windows Autopilot - Windows Deployment
Content Source: windows/deployment/windows-autopilot/windows-autopilot.md
Product: w10
Technology: windows
GitHub Login: @greg-lindsay
Microsoft Alias: greglin

Yes, Microsoft has a video hosting service. I'll try to get this video uploaded for you.

Hi Greg, I appreciate that. Thanks for the prompt reply.

How to find the user belongs to Microsoft.

@officedocsbot assign @jvsam

@officedocsbot assign @greg-lindsay

I don't own this video and the owners haven't replied. I imagine it'll have to wait until after the new year for people to get back and respond.

This is still taking a long time, partly because I have to wait for replies from someone else and partly because I'm working on this on the back burner.  I made a little more progress today and hopefully it won't be too much longer until the video is copied to a place other than YouTube.

No worries, no rush on this its just nice to know you're able to host
it somewhere else for those of us that can't use YouTube at work.

Sorry this took so long. The video is now also hosted here: https://www.microsoft.com/en-us/videoplayer/embed/RE4ATOx
I will add a link to this on the page also.


Add support for Azure OpenAI

Hi, I would like to try our the tool but we are using Azure OpenAI, which requires a few settings to change in the environment variables
such as
os.environ["OPENAI_API_TYPE"] = "azure"
os.environ["OPENAI_API_BASE"] = "https://<your-endpoint.openai.azure.com/"
os.environ["OPENAI_API_KEY"] = "your AzureOpenAI key"
os.environ["OPENAI_API_VERSION"] = "2023-05-15"
is it enough to add them in the Docker Compose file?

Thanks for your PR! Currently, adding support for other LLMs isn't possible in ODIN. I've raised Issue #7 to track this feature request, so I'll close your PR in favor of that.


Added ouroboros-network-<IP_ADDRESS>

From https://github.com/intersectmbo/ouroboros-network at bdcb1c59ca376e02509b471fc87b465e509dcfb6

[x] checked with build-with-chap.sh in ouroboros-network


New reflection creation method example?

@fabrice-etanchaud prior releases of dbt-dremio created reflections by creating a secondary model with something like this in the contents:
-- depends_on: {{ ref('some_table') }}
{{
    config(
        materialized='raw_reflection',
        dataset='some_table'
    )
}}

It seems that in the latest release of the plugin, this method has changed per these details.  Could you please provide an example code beyond the final mention of the -- depends_on: {{ ref('some_table') }}?  I have found other mentions to reflections being mapped to the materializedview materialization setting; however, I have been unable to figure out the proper settings on my own.

and by the way, materializedview is the type of Relation associated with reflections (the database.schema.identifier model physical localisation) ;-)

Hi @fabrice-etanchaud
The above example is to me - as clear as mud...  not least beacuse for me, it doesn't work.
The SQL file  has only the -- depends on   line,  yet dbt seems to be trying to execute as a create or replace view for it.
The dbt project file has  (indented appropriately)
nbbulkref_raw:
+materialized: reflection
+reflection_type: raw
+arrow_cache: true
What have I missed?

Hi @spireitepaul , are you using the latest dbt-dremio version ?
Here is a working example :
file adherents_reflect.sql :
-- depends_on: {{ ref('adherents') }}
in dbt_project.yml :
adherents_reflect:
          +materialized: reflection
          +reflection_type: raw

and in the log :

/* {"app": "dbt", "dbt_version": "1.0.6", "profile_name": "consultation", "target_name": "unmanaged", "node_id": "model.lcbft_iso.adherents_reflect"} */

  alter dataset<EMAIL_ADDRESS>    create raw reflection "adherents_reflect"
    using
      display ("numero_adherent","est_nationalite_etrangere","est_resident_etranger","est_resident_fiscal_etranger","est_remedie","est_pli_non_distribuable","est_ppe","est_gda","est_fin","est_protege_juridiquement","mineur","taux_remediation","nationalite","protection_juridique","date_naissance","age","etat")

Hi @fabrice-etanchaud
Yep, using the latest,
I think maybe this is a nesting issue of some sort...
What level should this be indented to under the models
the tree as i understand it should be models//
so does the _reflect indent AT the same level as  or indent WITHIN
or somewhere else?
To me, it's not clear....   the docs don't seem to cover this, (I apologies if i missed it).

@spireitepaul , can you share your dbt_project.yml file ?

@spireitepaul , can you share your dbt_project.yml file ?
dbt_project.txt

So your 'cro' project files should be something like :
models/cro.sql (a view) or models/cro/*/*.sql (views)
models/nbbulkref_reflection.sql (a reflection)
I can find any indentation problem.
It seems to have a problem between the reflection location and the configuration.
if you don't move your reflection, maybe you should try the following config :
models:
  cro:
    # Config indicated by + and applies to all files under models/cro/
    # models/cro
    cro:
      +materialized: view
      +schema: no_schema
      nbbulkref:
        nbbulkref_reflection:
          +materialized: reflection
          +reflection_type: raw
          +arrow_cache: true

or try moving nbbulkref_reflection.sql to models/cro/
tell me if it helps

Hi @spireitepaul , did it help ?

Hi @spireitepaul , I am going to close this issue, feel free to contact me again if needed.
Best,

I know this issue is closed, but just leaving this here in case it's helpful.  The examples that @fabrice-etanchaud left show setting the parameters in your config file(s).  You can also do this in a model file.
/* models/some_model_reflection.sql */
-- depends_on: {{ ref('some_model') }}
{{
    config(
        materialized='reflection',
        arrow_cache='true',
        reflection_type='raw',
        dataset='some_model'
	)
}}

I have to do this for many raw sources to reduce latency with the upstream source, so wrote it into a macro:
/* Make a reflection object as a "model"

Arguments:
    tbl (str): the upstream model name that should be reflected

Environment Variable:
    DBT_REFLECTIONS: if 1, then reflections are run.  By default, this runs no reflections then. (keeps dev spaces clean)
*/

{% macro make_reflection(tbl) -%}
-- depends_on: {{ ref(tbl) }}
{{
    config(
        materialized='reflection',
        arrow_cache='true',
        reflection_type='raw',
        dataset=tbl,
        enabled= env_var('DBT_REFLECTIONS', '0') == '1'
	)
}}
{%- endmacro %}

Then my some_model_reflection.sql file just has {{ make_reflection('some_table') }}.  It might not work for everyone, but lets me keep things consolidated (and also only fire reflections on prod deployments using the env var).

@aaroncwhite
You sir are a hero...
Your stuff to put in the model file  was the answer for me...
-- depends_on: {{ ref('some_model') }}
{{
config(
materialized='reflection',
arrow_cache='true',
reflection_type='raw',
dataset='some_model'
)
}}
@fabrice-etanchaud   I had to move to a different project, but came back to this.
The .yml changes just don't work for me, having been over them with a fine tooth comb.
What dbt-dremio was doing was trying to issue a create/replace view statement on the reflection.sql model, which is clearly wrong.  I couldn't get past it...


Overload series.rolling.apply()

@densmirn sorry not saying it in the first comment. Could you please also add tests with args and skip it? So we wouldn't forget to fix it

@densmirn sorry not saying it in the first comment. Could you please also add tests with args and skip it? So we wouldn't forget to fix it

Done.


Implement secondary Mantle interface

I updated this PR and added following three commits:

add and use finalizeStandalone and finalizeSecondary instead of finalize.
rename ReconcileAs{Standalone,Primary,Secondary} to reconcileAs{...}
add expiration process to reconcileAsSecondary.

@satoru-takeuchi I've fixed your points, so could you review this again?

@satoru-takeuchi I rebased and pushed my code. There should be no changes: https://github.com/cybozu-go/mantle/compare/601de02dd7d68d2a66a6f8612dda43a00bd2705f..024e131aea77e5fa7c266accbed41260606b6dca

Thank you!


*: add "getting started" example for LDAP

This is how I tested my changes for https://github.com/coreos/dex/pull/1021
Add a getting started guild using OpenLDAP. Hopefully others find this useful.
Not urgent.
cc @rithujohn191 @estroz

cc @dgonyeo you might be interested in the OpenLDAP standup script

comments addressed


Update the fallback anvil fork URL to use Cloudflare like viem does

Without this change, running the tests on a fresh checkout results in many failed tests with the following error repeated in the output:
Error: Anvil exited: thread 'main' panicked at /home/runner/work/foundry/foundry/crates/anvil/src/eth/backend/mem/mod.rs:238:39:
Failed to create genesis: GetAccount(0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266, (code: -32000, message: missing trie node 6be1147a8f956737d61a31ba05c807d2820b1559f30ca027abcc2dd5badbc544 (path ) state 0x6be1147a8f956737d61a31ba05c807d2820b1559f30ca027abcc2dd5badbc544 is not available, data: None)

Setting VITE_ANVIL_FORK_URL to point to an RPC from an Infura account resolves the issue, presumably because the current default, https://ethereum.publicnode.com is probably not an archive node. viem itself falls back to using a Cloudflare node, which also seems to solve the issue.
This pull request falls back to Cloudflare like viem does.

@niran
Thanks, queueing up this one too for the team to take a look. 🤝

@niran Thanks for this contribution!


Support for advanced Twitch API stuff

Summary
We should add some features of the Twitch API that are (in)directly related to chat.
Those include:

[x] check for a user to be a follower
[x] check for a user to be a subscriber (is this already shown in the tags attribute of an event? Needs verification to be working. Requires an application ID)

Those information are currently not fetched by the IRC stuff.

What do you think @joshuaskelly? Is this something worth implementing?
If so, what do you think about the current implementation in features/advanced-twitch-api-support?

I am aware of the fact, that urllib.request is not available in Python2, as our buddy Travis points out.

I think that is getting outside of the scope for twitchobserver. :worried:
I'm fairly confident there exist Python API's to handle this.

Yeah, I guess your're right. Besides that, one can check for a user to be a subscriber by looking at the tags component.
For followers one can keep a seperated list.


DRAFT: Add KFNBC feature flag and placeholder page.

This PR adds a KFNBC feature flag for the odh-dashboard-config CM.  In the backend the feature flag json file is checked against the CM value and disables/enables the notebook controller appearing in the components list. In frontend it enables/disables the jupyter-notebooks route.
The PR also add internal routes for components to allow linking within the dashboard from component tabs.
Also adds a basic nbc app card and route.

@maroroman @dlabaj is this still in draft?  Does @DaoDaoNoCode need this?

@cfchase I'll close this PR, I don't think it is needed anymore.


Missing _unsloth_temporary_saved_buffers

I'm getting the following error. I think it's probably because I'm running two training runs on the same machine which might try to create/delete the temporary file around the same time, so that the one that lags slightly behind can't find the temporary file anymore. I haven't validated that's what's happening, but hope that detail is helpful!
Traceback (most recent call last):
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/pdb.py", line 1723, in main
    pdb._runscript(mainpyfile)
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/pdb.py", line 1583, in _runscript
    self.run(statement)
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/bdb.py", line 598, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/jessy/projects/llm-objectives/sandbox/multistep_exp_faster.py", line 459, in <module>
    multistep_exp(args)
  File "/home/jessy/projects/llm-objectives/sandbox/multistep_exp_faster.py", line 238, in multistep_exp
    model = train_model(args, model, tokenizer, next_dataset, logdir, model_save_path)
  File "/home/jessy/projects/llm-objectives/sandbox/multistep_exp_faster.py", line 426, in train_model
    model.save_pretrained_merged(model_save_path, tokenizer, save_method = "merged_16bit")
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/site-packages/unsloth/save.py", line 980, in unsloth_save_pretrained_merged
    unsloth_save_model(**arguments)
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/site-packages/unsloth/save.py", line 632, in unsloth_save_model
    shutil.rmtree(temporary_location)
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/shutil.py", line 715, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/home/jessy/.miniconda3/envs/unsloth/lib/python3.10/shutil.py", line 713, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '_unsloth_temporary_saved_buffers'

@jlin816 Oh thanks for that!! Hmm I might leave the folder as is then! I'll add a check to not randomnly delete the folder :)

Thanks! Does this potentially cause any issues with running two jobs on the same machine (eg mixing up checkpoint data somehow)?

Hi @jlin816 hopefully the issue is fixed? Thanks!


Update Peer Dependency for graphile-worker

Fixes error when installing this package with newest graphile-worker version (0.13.0)

updated peer dependency graphile-worker to ^0.13.0
updated packages
updated version to 0.3.0

great, thank you

published on npm :raised_hands:


Feature query: stateful routes (module + viewport)

I've got code for stateful routes, meaning that if you do this
{ route: 'dashboard', name: 'dashboard', viewPorts: { 'default': { moduleId: 'dashboard', stateful: true } } }

the same instance of the dashboard module will be presented in the default viewport everytime a stateful route specifies it after the initial time. The view/viewmodel will remain loaded even when navigating away to another route, meaning that state such as list sorting, entered values, positions on google maps etc will be the same when coming back to it. It also means that the viewmodel will be able to listen to and act upon events from the event manager, signals and so on.
So to clarify, this is not caching for views/viewmodels, it's stateful views/viewmodels that are never unloaded.
If this is of interest to anyone else and has any chance of being accepted into the router I could put together a PR.

Is this something you want in? @EisenbergEffect @bryanrsmith

I like it!

@jwx Isn't this solved by making your view model a app singleton?

@EisenbergEffect see here

@davismj @EisenbergEffect Making the view model a singleton only preserves the view model state. State details not bound to the view model is lost, for example an unbound scroll position in a list or unbound position and zooming in a google map. Basically, this is completely cached views and so on.

Good info. I'll review again.
I'd like to pull this in out of the context of the optional and default view ports. This seems like a totally separate item.

I'd like to add in that making something singleton is not always viable. Sometime the element used at the leaf - route may end up being a real element in another context, thus making them singleton will not allow this scenario. My use case is I have a complex editor as a route, but then in another route, I reuse the element in a dialog.

Don't like the seeming hackiness of it. Kinda shoehorning.
you actually can achieve it with optional viewports now i think and a bit of css magic.

@Alexander-Taran See https://github.com/aurelia/router/pull/538.

@bigopon @jwx - any news on this or #538 ?


new release with updated macros

Hey, I was wondering if you could publish a new release to include the new macro code. I was having trouble figuring out why my code wasn't running, only to realise that I was missing metrics-macros + some other stuff which isn't mentioned on the main github readme.

Sounds like you're having a problem with existing crates that is unrelated to any of the changes that are currently unreleased. Or you're having a problem with using unreleased features by pulling in metrics as a Git dependency?
Either way, can you update this issue with your actual problem and the steps to reproduce it?

There is no bug, just confusion in regards to crate releases that I assume other newcomers might have also.  This was my process:

searched for metrics on crates.io and imported latest 0.21.1.
I then went to https://github.com/metrics-rs/metrics/tree/main/metrics/examples/basic and used the example counter syntax:

counter!("test_counter");

Received error

error: expected `,`
   --> src/main.rs:121:20
    |
121 |     let counter1 = counter!("test_counter");
    |                    ^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: this error originates in the macro `counter` (in Nightly builds, run with -Z macro-backtrace for more info)

After awhile, I realised that this version 0.21.1 requires a crate metric-macros that isn't listed anywhere on the main github page (as its not used anymore).
Just wondering when the next crates.io release will be? I am unable to directly use the repo in my cargo.toml due to some constraints.

Gotcha, yeah.
When will the next release be?
I'm sure when the next release will be, because there's a lot of stuff in flux at the moment that needs to be polished up, and other things slated to be included to get more of the breaking changes out of the way rather than having them be a slow trickle.
Compilation error based on using the example code
That error wasn't telling you that a crate is missing -- because you would have gotten an error like "cannot find counter! in current scope" or something to that effect. The error above is telling you that you can't use the macro in that way.
Mismatch between API in released crate and latest code in source control
Thanks for providing the steps, now I can see what your mindset was.
It seems like the missing link here was just that you didn't actually select the Git tag corresponding to the release you were using to see the source code that the release was specifically cut from.
Generally speaking, there's really nothing I can do about that (development is likely to always be in flux, so you have to know to find the tag/branch that represents the crate version you're using) but I would ultimately suggest using the crate documentation as it exists in docs.rs. I've put a lot of effort into that side of things, so the example code in the crate documentation should always be representative of the version of metrics (or the various other metric-*` crates) you're using.


[Question] Consume via Mavern dependency

Is there a way to consume the lib part of the project as a maven dependency? Do you publish releases on maven central or in any other public artifact repository?

The product has been published to the Maven repository.
Please use the following dependencies
dependencies {
    implementation("com.mammb:piecetable:0.5.0")
}


Add metadata builder support for Algorithm support

Add easy to use methods for adding <md:EncryptionMethod>, <alg:SigningMethod> and <alg:DigestMethod> elements to metadata.

Fixed in PR https://github.com/litsec/opensaml-ext/pull/51.


spec_helper: fix Timeout::Error/SystemExit handling.

These need to be manually caught and set otherwise they will not be retried by rspec-retry. This is particularly annoying and a cause of CI failures when tests timeout but are not retried.
add468b is pushed temporarily (and should not be merged) to attempt to reproduce the correct behaviour on a forced timeout.
Fixes https://github.com/Homebrew/brew/issues/8979

[x] Have you followed the guidelines in our Contributing document?
[x] Have you checked to ensure there aren't other open Pull Requests for the same change?
[x] Have you added an explanation of what your changes do and why you'd like us to include them?
[x] Have you written new tests for your changes? Here's an example.

[x] Have you successfully run brew tests with your changes locally?
[x] Have you successfully run brew man locally and committed any changes?

Confirmed from https://github.com/Homebrew/brew/pull/9004/checks?check_run_id=1326013554#step:13:38 that this is correctly retrying these examples now 🎉


net: provide RFC 6724 address sorting functionality

RFC 6724 describes a policy-based sorting algorithm for IP addresses that's to be applied by procedures like getaddrinfo.  Presumably Go's DNS resolver should provide support for this somehow, or at least the recommended default sorting policy.
On Linux with glibc, the system policy is configured via /etc/gai.conf.

If getaddrinfo will no longer be the default resolver in Go 1.5, then this feature needs to be a release blocker.  Address selection is a central feature of getaddrinfo which cannot simply be discarded.
Without getaddrinfo, the IPv4/IPv6 address selection is currently race-based.  This will cause Go clients to behave unpredictably when connecting to dual-stack names.

/cc @mikioh @mdempsky

It would help to understand why/how this is release blocking.  E.g., is simply implementing the RFC 6724 default sorting functionality sufficient?  Or do we need to support glibc /etc/gai.conf (and equivalents on other OSes) too?

Hard-coding the RFC 6724 rules would solve 99% of the problem, and I would not personally object to leaving the full gai.conf parser for post-1.5.
Pragmatically, the parts that really matter for Internet connectivity are:

Avoid unusable destinations.
If IPv6 would use a 6to4/Teredo/ULA source address, then prefer IPv4.
Otherwise prefer IPv6.

The standard approach is to probe each address using UDP connect and getsockname, to obtain reachability and source address information from the kernel.

I'd like to postpone handling this sort of, dual IP stack, issues to Go 1.6. The upcoming Go 1.5 contains @pmarks-net's Happy Eyeballs implementation and it could provide fairly better TCP connection setup than released versions on various environments such as IPv4-only, IPv6-only, dual IP stack w/o translation techniques or with XLAT, MAP-T or DS-lite, MAP-E-like translation techniques. Performance and connectivity improvements are desirable. But before doing that, I think we need to address issues in the real world carefully.
Random thoughts:

We don't need to consider 6to4 any more. See https://tools.ietf.org/html/rfc7050
Both http://www.ietf.org/mail-archive/web/v6ops/current/msg22275.html and http://www.ietf.org/mail-archive/web/v6ops/current/msg22290.html threads are pretty interesting

@mikioh I don't understand what point you're trying to convey.  Are you suggesting that we leave things as-is, and give users a resolver with random address preference in 1.5?
Real world networks are designed under the assumption that endpoints will attempt to follow the RFCs; that's why the RFCs exist.
I reiterate that random address preference is a Really Bad Idea; Happy Eyeballs would do a reasonable job of hiding the ensuing brokenness, but that isn't enabled by default (and probably shouldn't be, until dialTCP supports cancellation.)

RFC 7050 relates to NAT64, which is unrelated to 6to4.  As for why 6to4 source addresses must be de-preferred, this site has relevant data: http://www.fud.no/ipv6/
Back before the first World IPv6 Day, lots of users had broken 6to4 connectivity and preferred IPv6, which resulted in long timeouts when connecting to dual-stack hostnames.  The problem largely disappeared in 2011, when the last of the major browsers fixed their address selection rules.
Since the data is browser-based, we no longer know how bad the problem would be if those fixes were dropped.  It might be better, might be worse, but nobody is looking anymore.

Ouch, I meant https://tools.ietf.org/html/rfc7526.

@pmarks-net,
I'm not convinced of implementing non-configurable destination address selection described in RFC 6724 would solve 99% of the problem, because it would probably interfere with user's DNS recursor stuff, and when #11450 would be a solution for it there's no need to use builtin DNS stub resolver, but I may be wrong.

RFC 7526 does not imply that address selectors can delete their 6to4 rules; on the contrary, it means that de-preferring 6to4 is even more important, because wherever 6to4 has been deployed, it is now more likely to provide broken connectivity.  Since most other clients follow RFC 3484/6724, this brokenness will predominantly affect Go clients.
Could you clarify how RFC 6724 would interfere with DNS recursion?

If I understand correctly RFC 6724 leaves issues on various site-specific implementations to site operators. If a site that implements DNS full service resolver stuff prefers IPv4 rather than IPv6, non-configurable address selection policy might annoy the site operator. For example, what if they use a plain-old DNS round-robin as a last resort? This sort of issue happens not only for an IPv6-to-IPv4 fallback case but for an IPv6-to-IPv6 fallback case; what if a node has multiple prefixes and one of the prefixes is not global routable?
I agree on fixing this issue is necessary, and personally hope it comes with some interface for site operators.


Don't export @set

OhMyThreads.@set is pretty niche functionality and it clashes with Accessors.@set. Currently it's exported, would it be okay to not export it?

It's not at all niche if you use the @tasks macro API. I would recommend that you only explicitly import/using the bits that you need rather then just doing using OhMyThreads.
While the simple using X is, unfortunately, very common in Julia codes, it is quite generally an anti-pattern that doesn't scale well. (Any export that a package might add over time could break your code, because clashes like the one you mention here could occur.)


AppInsightsLogger malformed XML

Ensure "Enable Just My Code" is not enabled
Repro Steps

Create an instance of the AppInsightsLogger:

new AppInsightsLogger("sdfsdf");

Actual Behavior
System.Xml.XmlException occurred
  HResult=0x80131940
  Message=Name cannot begin with the ' ' character, hexadecimal value 0x20. Line 1, position 136.
  Source=System.Xml
  StackTrace:
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)

AppInsightsLogger constructor has a malformed XML constant:
XDocument.Parse("<?xml version=\"1.0\" encoding=\"utf-8\"?><ApplicationInsights xmlns = \"http://schemas.microsoft.com/ApplicationInsights/2013/Settings\"></ ApplicationInsights >");

Expected Behavior
No First Chance Exception
Version Info
SDK Version :  <package id="Microsoft.ApplicationInsights.SnapshotCollector" version="1.1.0" targetFramework="net47" />
.NET Version :	4.7
How Application was onboarded with SDK(VisualStudio/StatusMonitor/Azure Extension) : N/A
OS : N/A
Hosting Info (IIS/Azure WebApps/ etc) : N/A

What is AppInsightsLogger?

A type used by the AI SnapshotCollector. Reference that package & create an instance, without just my code, break on 1st chance.

@pharring can you please take a look?
Also where can I file bugs on it? Why snapshot collector exposes ILogger implementation as a public class? And uses AppInsights instead of ApplicationInsights.

Repro:

Add Microsoft.ApplicationInsights.SnapshotCollector

[TestMethod]
public void TrackTrace()
{
    using (new FirstChanceExceptionObserver())
    {
        var telemetryClient = new TelemetryClient(_telemetryConfiguration);
        telemetryClient.TrackTrace("Test");
        telemetryClient.Flush();
    }
}

With:
public sealed class FirstChanceExceptionObserver : IDisposable
{
    readonly ConcurrentBag<Exception> _exceptions = new ConcurrentBag<Exception>();

    public FirstChanceExceptionObserver() => AppDomain.CurrentDomain.FirstChanceException += OnFirstChanceException;

    public void Dispose()
    {
        AppDomain.CurrentDomain.FirstChanceException -= OnFirstChanceException;
        CheckForExceptions();
    }

    void CheckForExceptions()
    {
        switch (_exceptions.Count)
        {
            case 0:
                return;
            case 1:
                throw _exceptions.First();
            default:
                throw new AggregateException(_exceptions);
        }
    }

    void OnFirstChanceException(object sender, FirstChanceExceptionEventArgs firstChanceExceptionEventArgs) => _exceptions.Add(firstChanceExceptionEventArgs.Exception);
}

This appears to be related to a larger issue #682 of first chance exceptions being introduced during initialization.


Websocket Support

Checklist

[x] I've searched for similar feature requests.

Enhancement request
Allow for simple websocket testing
Problem it solves
E.g. “I'm always frustrated when […]”, “I’m trying to do […] so that […]”.
I am building a websocket server and would like an easy way to connect and send messages to it. Nothing too complex.
Basic features would be:

Sending Messages
Receiving and displaying messages

Additional information, screenshots, or code examples
Here is a link to a website that does something like this https://livepersoninc.github.io/ws-test-page/
…

Would really need this one too!

This would be fantastic!

That would be a great feature. Thank you.

sounds great, would be willing to help with this if needed

I think we should be aiming at something similar to websocat

Ok, so right now I have a general implementation where you can send one message or multiple messages until you close it with exit().
Example call for now is:
https WSS ws://<IP_ADDRESS>:8000/ws
Now I would ask for some advice/guideance on how the maintainers would like the implementation to be plugged in.
As for now, I have plugged it in the collect_messages() where I read if the method is WSS and then branch it out to go to a function to create a websocket connection and then another function that reads user input in a while loop sends it and waits for another message until the user types in exit(). In the meantime we have a thread that sends pings to keep the connection alive. Now I would have to implement models and leverage the argsparser better.
The question for now is, do I try to plug it into the present client and request handler or do we separate it as websockets are handled differently anyway.
This is the branch https://github.com/valentin994/cli/tree/websocket-support
It is quite crude for now as I am not sure what way we want to go with this.

I wrote a websocket plugin named httpie-websockets，https://github.com/belingud/httpie-websockets, that allows you to use http ws://localhost:8765/ establish a websocket connection, similar to websocat, listening to input and displaying messages. Pretty stable now. Welcome to try.


Only try to deploy from Python 3.7

This avoids having all builds failing because only the first one
that gets to the build will actually deploy it

Coverage remained the same at 99.139% when pulling 189e0765c28b58a36ccd87fd95bc43725ddeb7ae on nicoddemus:deploy-fix into e5c11f1aa6d9bcf6b73cfee93d810027aa898e8a on Bachmann1234:master.

ill live coveralls, but thanks for warning me :-p

Thank you!


Cleanup initial sql statements:

Check if tables and indexes exist
Remove ref to non-existent PeerImps table

@miguelportilla @scottschurr

Obviously correct, please test.


Pre-allocation of memory on the object store.

When chunks are created they pre-allocate memory on the object store, this is undesired behaviour and not required as object stores do not require memory pre-allocation.

Upon further investigation, this only happens when trying to access chunks that do not exist i.e after the dataset has been cleared.  Therefore this issue has been closed.


arch: rename arch special function name

Summary
Rename arch special function name to arch_xxxxx.c
Impact
NA
Testing
sim:nsh

@zyfeier Why? What is the motivation AND justification for this change?

@zyfeier Why? What is the motivation AND justification for this change?

@davids5 In the NuttX documentation we can see " An interface which is unique to a certain microprocessor should be prefixed with the name of the microprocessor". So, we modify arch special interface name to meet the requirements.

This is the continue work from @patacongo :
https://github.com/apache/incubator-nuttx/pull/926
https://github.com/apache/incubator-nuttx/pull/929
https://github.com/apache/incubator-nuttx/pull/930
https://github.com/apache/incubator-nuttx/pull/935
https://github.com/apache/incubator-nuttx/pull/937
https://github.com/apache/incubator-nuttx/pull/940
https://github.com/apache/incubator-nuttx/pull/948
https://github.com/apache/incubator-nuttx/pull/958
https://github.com/apache/incubator-nuttx/pull/958
https://github.com/apache/incubator-nuttx/pull/961

Should add "breaking change" label just in case any out-of-tree code calls arch-specific functions?

Should add "breaking change" label just in case any out-of-tree code calls arch-specific functions?

add

@davids5 and @pkarashchenko the patch is ready for review, please ignore the first three patch which belong to https://github.com/apache/incubator-nuttx/pull/7599 but this PR depends on them.


Use rust bindgen for reading/writing the PostgreSQL control file.

My findings so far on the use of transmute:
Since ControlFileData contains bools, transmute from a byte array is unsound (if the byte contains anything but 0 or 1, undefined behavior will result).  If we expand our use of bindgen further, I expect we'll discover more ways to trigger dangerous things. So I'd prefer to avoid transmute on anything auto-generated: it's not obvious what types are in there, now or in the future.
I hoped there would be a way to attach #[derive(Serialize, Deserialize)] to the generated structs, but this doesn't seem possible right now. There are a few crates out there that also have this problem, and they seem to be using regex workarounds (example).
I also looked around at some of the "safe transmute" crates and RFC 2981. While there is a lot of potential there, I didn't find anything that's able to solve this problem now.
Some possible ways forward:
a) Continue with transmute until we can figure out how to do something better. Perhaps add a big warning comment, to remind us to fix this before it gets out of hand.
b) Go back to hand-written structs. We can replace the encode/decode functions with #[derive(Serialize, Deserialize)] if we like.
c) Build a procedural macro that can apply #[derive(Serialize, Deserialize)].
d) Run the bindgen output through a text regex to insert Serialize, Deserialize in the right place.
(d) seems crude, but it's so much less work than (c) that I think it's probably the right way forward, unless we expect to only have a few C structs imported with bindgen, then handwritten is probably fine.

My findings so far on the use of transmute:
Since ControlFileData contains bools, transmute from a byte array is unsound (if the byte contains anything but 0 or 1, undefined behavior will result). If we expand our use of bindgen further, I expect we'll discover more ways to trigger dangerous things. So I'd prefer to avoid transmute on anything auto-generated: it's not obvious what types are in there, now or in the future.

Perhaps we could map Postgres bool to Rust's u8 instead of bool. Not sure how to tell bindgen to do that.

I hoped there would be a way to attach #[derive(Serialize, Deserialize)] to the generated structs, but this doesn't seem possible right now. There are a few crates out there that also have this problem, and they seem to be using regex workarounds (example).

Yeah, that would've been nice.

I also looked around at some of the "safe transmute" crates and RFC 2981. While there is a lot of potential there, I didn't find anything that's able to solve this problem now.
Some possible ways forward:
a) Continue with transmute until we can figure out how to do something better. Perhaps add a big warning comment, to remind us to fix this before it gets out of hand.
b) Go back to hand-written structs. We can replace the encode/decode functions with #[derive(Serialize, Deserialize)] if we like.
c) Build a procedural macro that can apply #[derive(Serialize, Deserialize)].
d) Run the bindgen output through a text regex to insert Serialize, Deserialize in the right place.
(d) seems crude, but it's so much less work than (c) that I think it's probably the right way forward, unless we expect to only have a few C structs imported with bindgen, then handwritten is probably fine.

I'm inclined to do a) for now, and revisit this and perhaps switch to d) if we need many more of these structs. If we're lucky, this gets addressed in bindgen in the meanwhile.
We have a few hand-written structs in the WAL decoding routines that mirror Postgres structs, but those are fairly short and we only need to read them, not write. But if we had a more general solution, would be nice to use it for those too.

Perhaps we could map Postgres bool to Rust's u8 instead of bool. Not sure how to tell bindgen to do that.

I looked for a way to tell bindgen to remap bool to u8 and didn't find one. I think we'd have to write something to post-process the entire bindgen output file. Or trick it by patching the input file with #define bool uint8_t or something like that.

Do we want to be able to convert &[u8] to &ControlFileData (and vice versa)?
Right now it looks like there's a copy happening, unless the compiler can optimize it away.

There's an (undocumented?) standard practice in Rust, which is that whenever you use unsafe, you add a SAFETY comment explaining why this code can't cause bad things to happen. An example from the standard library:
        // SAFETY: the align is guaranteed by Rust to be a power of two and
        // the size+align combo is guaranteed to fit in our address space. As a
        // result use the unchecked constructor here to avoid inserting code
        // that panics if it isn't optimized well enough.
        unsafe { Layout::from_size_align_unchecked(size, align) }


Ribosomal RNA operon copy number correction / rrnDB support

Hi,
maybe it would be interesting/useful to add support for things like the rrnDB. This is often combined with using absolute abundance measures (see #4) to get estimated genome copy numbers instead of rRNA operon copy numbers, which is a better proxy for cell numbers than just the operon copies.
Possible helpers could be:

a function to (download and) match the rrnDB to the row data, at the best/highest/fixed taxonomic level
a function to calculate corrected copy numbers (probably in combination with #4) to get absolute genome copy numbers.

Best,
Bela

Thanks! The procedure should be probably tested before deploying it into wider use. Hence, a Rmarkdown vignette (for instance in this or in OMA) might be a good start for the discussion on how to convert it into a R pkg tool. Contributions also very welcome.

I think this has to implemented elsewhere, since mia can provided the tools for applying a correction, but interlinking with datasources is out of scope IMO.
Maybe and adjacent package would best.


Renaming containers doesn't update dnsdock names

Once a container is launched with a particular name, it sticks until the container is removed:
$ docker run --name one -d httpd
ee030eb30dc69e04eade85dd535707b8d768fc42dd4524bcb37721873c1b4868
$ docker logs dnsdock 2>&1 | grep ee03
2015/06/25 15:41:11 Added service: ee030eb30dc69e04eade85dd535707b8d768fc42dd4524bcb37721873c1b4868 {one httpd <IP_ADDRESS> -1 []}
$ host one.httpd.dnsdock
one.httpd.dnsdock has address <IP_ADDRESS>
$ docker rename one two
$ docker logs dnsdock 2>&1 | grep ee03
2015/06/25 15:41:11 Added service: ee030eb30dc69e04eade85dd535707b8d768fc42dd4524bcb37721873c1b4868 {one httpd <IP_ADDRESS> -1 []}
$ host two.httpd.dnsdock
Host two.httpd.dnsdock not found: 3(NXDOMAIN)
$ host one.httpd.dnsdock
one.httpd.dnsdock has address <IP_ADDRESS>

Perhaps this is because Docker's events API doesn't report on name changes?

Solved


Enlarge HIPSCAT_ID_HEALPIX_ORDER

Currently HIPSCAT_ID_HEALPIX_ORDER=19 which corresponds to ~0.4'', which is larger than LSST pixel size (0.2''), similar to Gaia and DECaPS angular resolution (0.4''-0.5''), and significantly larger than angular resolution of the space missions like HST  and JWST (~0.1'').
I understand that currently we'd like to keep _hipscat_index to fit 64bit and have counter large enough LSST sources, which limits us from using HIPSCAT_ID_HEALPIX_ORDER=29 or something like that. However, I believe that something like 23-24 would practically work well.

Ok, probably this was a bad idea - I was thinking to use _hipscat_index as an object ID, which is an anti-pattern.
We may also think about doing an opposite - to make counter part larger. However, currently I see no practical reason for that.


wavenet tts not working

Hello,
I am on a fresh install of 2.4.20 on a fresh install of raspbian buster, and I've just added credentials and configuration for the wavenet tts. However it always falls back to espeak.
This is what I get in the logs:
Apr 23 11:28:10 voicebase rhasspy[8163]: DEBUG:GoogleWaveNetSentenceSpeaker:Trying credentials at /home/pi/.config/rhasspy/profiles/de/tts/googlewavenet/credentials.json
Apr 23 11:28:10 voicebase rhasspy[8163]: DEBUG:GoogleWaveNetSentenceSpeaker:Calling Wavenet (lang=de-DE, voice=de-DE-Wavenet-C, gender=FEMALE, rate=22050)
Apr 23 11:28:10 voicebase rhasspy[8163]: ERROR:GoogleWaveNetSentenceSpeaker:speak
Apr 23 11:28:10 voicebase rhasspy[8163]: Traceback (most recent call last):
Apr 23 11:28:10 voicebase rhasspy[8163]:   File "/home/pi/rhasspy/rhasspy/tts.py", line 643, in in_ready
Apr 23 11:28:10 voicebase rhasspy[8163]:     self.wav_data = self.speak(message.sentence, voice, language_code)
Apr 23 11:28:10 voicebase rhasspy[8163]:   File "/home/pi/rhasspy/rhasspy/tts.py", line 742, in speak
Apr 23 11:28:10 voicebase rhasspy[8163]:     from google.cloud import texttospeech
Apr 23 11:28:10 voicebase rhasspy[8163]: ImportError: cannot import name 'texttospeech' from 'google.cloud' (unknown location)

Not sure what could be wrong here?

Same error, I too I'm working on 2.4.20, seems that the tts.py can't import google tts client libraries
[DEBUG:2242291] GoogleWaveNetSentenceSpeaker: Falling back to PicoTTSSentenceSpeaker [ERROR:2242289] GoogleWaveNetSentenceSpeaker: speak Traceback (most recent call last):   File "/usr/share/rhasspy/rhasspy/tts.py", line 643, in in_ready     self.wav_data = self.speak(message.sentence, voice, language_code)   File "/usr/share/rhasspy/rhasspy/tts.py", line 742, in speak     from google.cloud import texttospeech ImportError: cannot import name 'texttospeech' from 'google.cloud' (unknown location) [DEBUG:2242287] GoogleWaveNetSentenceSpeaker: Calling Wavenet (lang=en-US, voice=Wavenet-C, gender=MALE, rate=22050) [DEBUG:2242286] GoogleWaveNetSentenceSpeaker: Trying credentials at /profiles/en/tts/googlewavenet/Mywai-f573c55b856b.json

Same here... seems google tts module is missing?

Well at least when trying to install it by hand it says its already installed:
pi@voicebase:~/rhasspy $ source .venv/bin/activate
(.venv) pi@voicebase:~/rhasspy $ pip3 install google-cloud-speech
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Requirement already satisfied: google-cloud-speech in ./.venv/lib/python3.7/site-packages (1.3.1)
Requirement already satisfied: google-api-core[grpc]<2.0.0dev,>=1.14.0 in ./.venv/lib/python3.7/site-packages (from google-cloud-speech) (1.17.0)
Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (2.22.0)
Requirement already satisfied: google-auth<2.0dev,>=1.14.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (1.14.1)
Requirement already satisfied: setuptools>=34.0.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (40.8.0)
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (1.51.0)
Requirement already satisfied: six>=1.10.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (1.14.0)
Requirement already satisfied: pytz in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (2019.3)
Requirement already satisfied: protobuf>=3.4.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (3.11.3)
Requirement already satisfied: grpcio<2.0dev,>=1.8.2; extra == "grpc" in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (1.28.1)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (1.25.9)
Requirement already satisfied: idna<2.9,>=2.5 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (2<IP_ADDRESS>)
Requirement already satisfied: pyasn1-modules>=0.2.1 in ./.venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (0.2.8)
Requirement already satisfied: rsa<4.1,>=3.1.4 in ./.venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (4.0)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in ./.venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (4.1.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in ./.venv/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-speech) (0.4.8)
(.venv) pi@voicebase:~/rhasspy $

forgot to clarify, i'am on the rhasspy docker image.
@patrickjane  if you are running rhasspy in a docker container too: did you log into this container?
perhaps @Romkabouter can help? ;)

@linuxlurak nope I'm on a venv installation.

I have got the same issue, I'll see if I can fix it

I had not noticed it, because playing from cache works fine

Seems like there was a missing dependency. After pip install --upgrade google-cloud-texttospeech it still doesnt work, however the error is different in the logs:
May  8 09:39:06 calypso rhasspy[15950]: ERROR:GoogleWaveNetSentenceSpeaker:speak
May  8 09:39:06 calypso rhasspy[15950]: Traceback (most recent call last):
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
May  8 09:39:06 calypso rhasspy[15950]:     return callable_(*args, **kwargs)
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
May  8 09:39:06 calypso rhasspy[15950]:     return _end_unary_response_blocking(state, call, False, None)
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
May  8 09:39:06 calypso rhasspy[15950]:     raise _InactiveRpcError(state)
May  8 09:39:06 calypso rhasspy[15950]: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
May  8 09:39:06 calypso rhasspy[15950]: #011status = StatusCode.INVALID_ARGUMENT
May  8 09:39:06 calypso rhasspy[15950]: #011details = "Request contains an invalid argument."
May  8 09:39:06 calypso rhasspy[15950]: #011debug_error_string = "{"created":"@1588927146.806035885","description":"Error received from peer ipv4:<IP_ADDRESS>:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Request contains an invalid argument.","grpc_status":3}"
May  8 09:39:06 calypso rhasspy[15950]: >
May  8 09:39:06 calypso rhasspy[15950]: The above exception was the direct cause of the following exception:
May  8 09:39:06 calypso rhasspy[15950]: Traceback (most recent call last):
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/rhasspy/tts.py", line 643, in in_ready
May  8 09:39:06 calypso rhasspy[15950]:     self.wav_data = self.speak(message.sentence, voice, language_code)
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/rhasspy/tts.py", line 761, in speak
May  8 09:39:06 calypso rhasspy[15950]:     response = client.synthesize_speech(synthesis_input, voice_params, audio_config)
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/cloud/texttospeech_v1/gapic/text_to_speech_client.py", line 322, in synthesize_speech
May  8 09:39:06 calypso rhasspy[15950]:     request, retry=retry, timeout=timeout, metadata=metadata
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in __call__
May  8 09:39:06 calypso rhasspy[15950]:     return wrapped_func(*args, **kwargs)
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
May  8 09:39:06 calypso rhasspy[15950]:     on_error=on_error,
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
May  8 09:39:06 calypso rhasspy[15950]:     return target()
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
May  8 09:39:06 calypso rhasspy[15950]:     return func(*args, **kwargs)
May  8 09:39:06 calypso rhasspy[15950]:   File "/home/pi/rhasspy/.venv/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
May  8 09:39:06 calypso rhasspy[15950]:     six.raise_from(exceptions.from_grpc_error(exc), exc)
May  8 09:39:06 calypso rhasspy[15950]:   File "<string>", line 3, in raise_from
May  8 09:39:06 calypso rhasspy[15950]: google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.

Continuing to investigate ...

Okay, found the issue. I was giving de-DE-Wavenet-C as voice (this is shown at the google website), but rhasspy adds an additional de-DE in front of it, making an invalid argument. I've fixed it, and now wavenet TTS works.
This is the relevant part in profile.json:
    "text_to_speech": {
        "system": "wavenet",
        "wavenet": {
            "cache_dir": "tts/googlewavenet/cache",
            "credentials_json": "tts/googlewavenet/credentials.json",
            "voice": "Wavenet-C",
            "url": "https://texttospeech.googleapis.com/v1/text:synthesize",
            "fallback_tts": "picotts"
        }
    },

Good find, so I think the docker should be updated

I am not using docker, I did the venv installation.

I am not using docker, I did the venv installation.

I logged into the docker image, installed pip and then the google-cloud-texttospeech library and then restarted rhasspy but I still have the same error
ImportError: cannot import name 'texttospeech' from 'google.cloud' (unknown location)
Is there a proper way to install thos python libraries in the docker image?

Okay so I remember that some weeks ago I already tried to fix this, and installed some google package, but still it didnt work, so I stopped investigating. Maybe theres a second package missing.
Lets compare:
(.venv) pi@calypso:~/rhasspy $ pip list | grep google
google-api-core           1.17.0
google-auth               1.14.1
google-cloud-speech       1.3.1
google-cloud-texttospeech 1.0.1
googleapis-common-protos  1.51.0

google-api-core           1.17.0
google-auth               1.14.2
google-cloud-speech       1.3.2
google-cloud-texttospeech 1.0.1
googleapis-common-protos  1.51.0
Pretty the same for my docker image.
The google-cloud-speech library was missing and I manually installed it but isn't relevant with the issue

Okay so I just did a complete new install, and for me it was fixed after manually installing google-cloud-texttospeech.
(.venv) pi@calypso:~/rhasspy $ pip install --upgrade google-cloud-texttospeech
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting google-cloud-texttospeech
  Using cached google_cloud_texttospeech-1.0.1-py2.py3-none-any.whl (50 kB)
Requirement already satisfied, skipping upgrade: google-api-core[grpc]<2.0.0dev,>=1.14.0 in ./.venv/lib/python3.7/site-packages (from google-cloud-texttospeech) (1.17.0)
Requirement already satisfied, skipping upgrade: google-auth<2.0dev,>=1.14.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (1.14.3)
Requirement already satisfied, skipping upgrade: requests<3.0.0dev,>=2.18.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (2.22.0)
Requirement already satisfied, skipping upgrade: googleapis-common-protos<2.0dev,>=1.6.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (1.51.0)
Requirement already satisfied, skipping upgrade: six>=1.10.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (1.14.0)
Requirement already satisfied, skipping upgrade: setuptools>=34.0.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (40.8.0)
Requirement already satisfied, skipping upgrade: protobuf>=3.4.0 in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (3.11.3)
Requirement already satisfied, skipping upgrade: pytz in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (2020.1)
Requirement already satisfied, skipping upgrade: grpcio<2.0dev,>=1.8.2; extra == "grpc" in ./.venv/lib/python3.7/site-packages (from google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (1.28.1)
Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in ./.venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (0.2.8)
Requirement already satisfied, skipping upgrade: rsa<4.1,>=3.1.4 in ./.venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (4.0)
Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in ./.venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (4.1.0)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (2.8)
Requirement already satisfied, skipping upgrade: chardet<3.1.0,>=3.0.2 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (3.0.4)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (2<IP_ADDRESS>)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.venv/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (1.25.9)
Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in ./.venv/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.14.0->google-api-core[grpc]<2.0.0dev,>=1.14.0->google-cloud-texttospeech) (0.4.8)
Installing collected packages: google-cloud-texttospeech
Successfully installed google-cloud-texttospeech-1.0.1

So after installation I had:
(.venv) pi@calypso:~/rhasspy $ pip list | grep google
google-api-core          1.17.0
google-auth              1.14.3
google-cloud-speech      1.3.1
googleapis-common-protos 1.51.0

In this, google-cloud-texttospeech was missing (not google-cloud-speech).

I am using the hassio addon and have the same problems with wavenet tts.
GoogleWaveNetSentenceSpeaker: Falling back to EspeakSentenceSpeaker
[ERROR:87465816] GoogleWaveNetSentenceSpeaker: speak
Traceback (most recent call last):
File "/usr/share/rhasspy/rhasspy/tts.py", line 643, in in_ready
self.wav_data = self.speak(message.sentence, voice, language_code)
File "/usr/share/rhasspy/rhasspy/tts.py", line 742, in speak
from google.cloud import texttospeech
ImportError: cannot import name 'texttospeech' from 'google.cloud' (unknown location)
Is there a solution for the addon too?

If have found the cause of this issue.
Back on 7th of december this commit:
https://github.com/synesthesiam/rhasspy/commit/080ba2e97fd102eab7f1165ddef3fcaa07c9191e
It set google from true to false, causing to skip the install of the Google TTS.
I will create a PR today or tomorrow

PR is merged, can you retry?


chore: release main

:robot: I have created a release beep boop
dev-dependencies-client: 1.1.1
1.1.1 (2023-11-10)
Bug Fixes

update client dependencies to use common (0ac93c2)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/aversini/dev-dependencies/releases/tag/dev-dependencies-client-v1.1.1 :sunflower:


Autocomplete install does not work in zsh

Q
A

Issue Type
Bug

Deployer Version
5.1.3

Local Machine OS
Ubuntu 14.04.5 LTS

Remote Machine OS
N/A

Description
autocomplete install does not work. I guess I am missing some package, but maybe script should give me some clue what I am missing?
Steps to reproduce
$ dep autocomplete --install > ~/.deployer_completion && echo "source ~/.deployer_completion" >> ~/.zshrc
$ source ~/.zshrc
Output log
/home/vytautas/.deployer_completion:47: command not found: complete

autoload bashcompinit
bashcompinit

autoload bashcompinit
bashcompinit
source ~/.zshrc

after:
_deployer:4: command not found: _get_comp_words_by_ref
_deployer:7: bad option: -t
_deployer:38: command not found: __ltrim_colon_completions

I think it's a bug then. Please check this project to https://github.com/bamarni/symfony-console-autocomplete
Deployer's autocomplete based on it.

looks like its working just had to add to .zshrc
eval "$(/home/vytautas/.composer/vendor/bin/symfony-autocomplete)"

instead of
eval "$(symfony-autocomplete)"

but not sure this is related

okay added  composer bin path to $PATH and now works eval "$(symfony-autocomplete)" but deployer still does not

also symfony-autocomplete works without
autoload bashcompinit
bashcompinit

Really strange, because they are really same. Try this:
eval "$(cat  ~/.deployer_completion)"

no, does not help same output

Can't tell what's wrong then)

by quick look at bash scripts they are different maybe they updated something? also symfony-autocomplete has different templates for bash and zsh but deployer has just one template? no?

okay now I see deployer uses bash template from that project even I am using zsh I guess that's the problem

also symfony-autocomplete has different templates for bash and zsh
No, only bash impl

@antonmedv when Iinstalled symfony-autocomplete it created with this https://github.com/bamarni/symfony-console-autocomplete/blob/master/resources/zsh/default.php
and when I installed deployer it used this:
https://github.com/bamarni/symfony-console-autocomplete/blob/master/resources/bash/default.php
from my error you can see deployer version tries to use _get_comp_words_by_ref this only exists in bash template
_deployer:4: command not found: _get_comp_words_by_ref
also I don't need
autoload bashcompinit
bashcompinit

because https://github.com/bamarni/symfony-console-autocomplete/blob/master/resources/zsh/default.php does not have this line https://github.com/bamarni/symfony-console-autocomplete/blob/master/resources/bash/default.php#L46

Ok, thanks. I see. Well) I think it's a time for PR? Maybe optional param to install? --install zsh

What's the outcome of this issue? It's still open, but the problem still exists.

Yes, this issue still present. Can you make PR to fix it?

Is there any known workaround for this? I've tried things mentioned here and nothing actually fixes it. Also tried upgrading bash to v4 (MacOS) and no luck. Ref: https://github.com/dsifford/yarn-completion/issues/4
Thanks

Hey all — still getting this issue today. Clean install of deployer with recent zsh:
Used this to install:
dep autocomplete --install > ~/.deployer_completion && echo "source ~/.deployer_completion" >> ~/.zshrc

Completely closed CLI session then tried dep <tab>:
» dep …_deployer:4: command not found: _get_comp_words_by_ref "source ~/.deployer_completion"_deployer:type:7: bad option: -t
_deployer:38: command not found: __ltrim_colon_completions

I guess this was not fixed I forgot to create PR at the time. And I guess no one fixed this if it does not work. Not sure why @antonmedv closed if it's not fixed or maybe you just ran into a different issue.

It was completely redesigned in v7. Please, take a look there.

@njpanderson what version you use? I see docs changed since then https://github.com/deployphp/deployer/blob/57dea072e68979d79e43351a0a39931333e7fff2/docs/installation.md#autocomplete

Interesting. I'm just getting 6 from the phar, which seems to be the first method of installation as per the docs?
@svycka:
--- ~ » dep --version
Deployer 6.8.0

I see v7 is still in beta, any plans to release it this coming year?

@njpanderson I hope so because stable release didn't have any releases for almost  about one year I guess you can use v7 but maybe @antonmedv can say more about his plans

I have plans to release v7 as soon as I find some time)

MacOS 11.6
eval "$(dep autocomplete --shell bash) fails in .bashrc
dep autocomplete --shell bash fail on command line
Same error
The "--shell" option does not exist.

autocomplete [--install]

Ah I see. The docs are wrong at https://deployer.org/docs/7.x/installation
dep autocomplete --install | sudo tee /etc/bash_completion.d/deployer works as expected.

If I run dep autocomplete I get the following output:
dep autocomplete
To install Deployer autocomplete run one of the following commands:

# Bash (Ubuntu/Debian)

  dep autocomplete --install | sudo tee /etc/bash_completion.d/deployer

# Bash (Mac OSX with Homebrew "bash-completion")

  dep autocomplete --install > $(brew --prefix)/etc/bash_completion.d/deployer

# Zsh

  dep autocomplete --install > ~/.deployer_completion && echo "source ~/.deployer_completion" >> ~/.zshrc

# Fish

  dep autocomplete --install > ~/.config/fish/completions/deployer.fish

This is where I found the working command to add to bash.

I think this should be deleted and replaced as it is not really accurate and prevents the newer version from automatically poping in.


Event Simulator in carbon console doesn't seem to work

Description:
Simulating event using the event simulator is not working.
Suggested Labels:

Suggested Assignees:

Affected Product Version:
2.1.0
OS, DB, other environment details and versions:
Steps to reproduce:
Related Issues:

Fixed with GH-392


test: Disable toEntities Cluster and toEntities World for now

Both tests are not reliable enough yet.

This change is

test-me-please

Failed due to #5793

Closed in favor of #5792


SwaggerUI validation error after Heroku deployment

Added options to app.js to set the validation url in the swagger-ui-express package.

Coverage increased (+0.05%) to 79.703% when pulling 1fb7cb490095d2a4d290dcec7b0ecb1b39d0c2f2 on bugfix/swagger-heroku-error into<PHONE_NUMBER>67d7f24579880b6a9ecfb86fa1f794 on development.

Coverage increased (+0.05%) to 79.703% when pulling 1fb7cb490095d2a4d290dcec7b0ecb1b39d0c2f2 on bugfix/swagger-heroku-error into<PHONE_NUMBER>67d7f24579880b6a9ecfb86fa1f794 on development.

Something doesn't look quite right. There's a little image placeholder on the screen:

And clicking it takes you to an odd looking page:

In local mode, it needs to be set to null since it can't reach the validate anyways. That's the default behavior for the validator. Once it's deployed we can check to see if that actually worked for the URL but I though it was looking for an actual URL.

When it's deployed, the Url needs to be set to "https://autorenter-nodeexpress-api.herokuapp.com/swagger.json"
We have a validation error as well: https://online.swagger.io/validator/debug?url=https://autorenter-nodeexpress-api.herokuapp.com/swagger.json
{"schemaValidationMessages":[{"level":"error","domain":"validation","keyword":"anyOf","message":"instance failed to match at least one required schema among 2","schema":{"loadingURI":"http://json-schema.org/draft-04/schema#","pointer":"/properties/type"},"instance":{"pointer":"/definitions/Vehicle/properties/image/type"}}]}

Coverage increased (+0.1%) to 79.753% when pulling 52b2248589ecb30aceefa4733a57d24b8d256a73 on bugfix/swagger-heroku-error into 019c783b24bd0a2f9710270ea329a9b10ae0379e on development.

Coverage increased (+0.05%) to 79.703% when pulling e7dae36eb4d8d9e3025bdb59f34690219c581bf6 on bugfix/swagger-heroku-error into 019c783b24bd0a2f9710270ea329a9b10ae0379e on development.

Coverage increased (+0.05%) to 79.703% when pulling e7dae36eb4d8d9e3025bdb59f34690219c581bf6 on bugfix/swagger-heroku-error into 019c783b24bd0a2f9710270ea329a9b10ae0379e on development.


how to solve "invalid_request: ... redirect_uri is not valid.

when enter the user email, it redirect to the next page, but show:
We're unable to complete your request

invalid_request: The provided value for the input parameter 'redirect_uri' is not valid. The expected value is a URI which matches a redirect URI registered for this client application.

how to solve this?

You need a corporate 365 account it doesnt work on personal accounts


Removing cancelled uploads from server (on cancel action)

Hi
It would be really nice to have the option of removing cancelled uploads using some option like removeCancelled.
In our app, if the user closes the browser or the tab or reloads the page we take that as them saying they don't care about recovering what has been cancelled already (cancelling is an explicit act on their part.) Seeing all the cancelled uploads show up again after they've been cancelled and removed from the UI is a little confusing and cluttering.
When removeCancelled is set to true, the cancelled uploads should be DELETED from the server upon the user clicking on Cancel icon or the "cancel all" button.
I'd be happy to add that once I get comfortable with the code, but for now I thought I'd outline why it is not expected to see cancelled uploads show up upon page reload.
The behavior is there without enabling resume on add ... I'm not overwriting the add handler, just the fail handler.

I should add, this should work with chunked upload.

That's what the discard_aborted_uploads option is there for in the PHP upload handler.
Unfortunately, this option has to be disabled for chunked uploads to work, so you have to decide between:

Keeping partial uploads (for chunked uploads to work) or
Discarding them automatically.

What you could do is add a cron job to your application, which checks periodically for orphaned partial uploads.
But you would also have to add some kind of meta information for the files , to identify partial uploads, else the cron job would not be able to differentiate between regular files and partial leftovers.

And should not change the behavior for aborted uploads (uploads that are not explicitly cancelled by the user) Those should indeed show up when the page is reloaded.

Maybe a mental block on either end, but I was actually thinking that the call to DELETE the file from the server is issued when the user clicks "Cancel Uploads" or the cancel icon. In other words, it's initiated from the UI so you can simply issue a DELETE request to the server when the user Cancels the upload.
Am I missing something?

In other words, it is as simple as copying code from the "trash icon button" handler and adding it inside "if (fu.options.deleteCancelled) {... }"

The cancel action just invokes the abort method on the XHR upload, it doesn't call the DELETE request.
But you are right, requesting a DELETE after cancelling the upload fixes that issue.
I'll update the chunked upload docs.

Would be great if the doc update covers the "Cancel Uploads" (for all files) scenario not only the Cancel icon (for individual files) scenario. It's starting to look really good as something we'd use in our app. Thanks for that.

Already updated, covers all cases of aborting uploads.


[Feature] Support SLOCC per  a PR

https://www.dwheeler.com/sloccount/

Description: This is the home page of "SLOCCount", a set of tools for counting physical Source Lines of Code (SLOC) in a large number of languages of a potentially large set of programs. This suite of tools was used in my papers More than a Gigabuck: Estimating GNU/Linux's Size and Estimating Linux's Size to measure the SLOC of entire GNU/Linux distributions, and my essay Linux Kernel 2.6: It's Worth More! Others have measured Debian GNU/Linux and the Perl CPAN library using this tool suite.

:octocat: cibot: Thank you for posting issue #259. The person in charge will reply soon.

Test case with ./gst/ folder:

Totals grouped by language (dominant language first):
ansic:         6023 (94.39%)
cpp:            358 (5.61%)

Total Physical Source Lines of Code (SLOC)                = 6,381
Development Effort Estimate, Person-Years (Person-Months) = 1.40 (16.80)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 0.61 (7.30)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 2.30
Total Estimated Cost to Develop                           = $ 189,136
 (average salary = $56,286/year, overhead = 2.40).
SLOCCount, Copyright (C) 2001-2004 David A. Wheeler
SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL.
SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to
redistribute it under certain conditions as specified by the GNU GPL license;
see the documentation for details.
Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."

Test case with ./ci/ folder (TAOS-CI):

Totals grouped by language (dominant language first):
sh:            2447 (82.20%)
php:            322 (10.82%)
python:         208 (6.99%)

Total Physical Source Lines of Code (SLOC)                = 2,977
Development Effort Estimate, Person-Years (Person-Months) = 0.63 (7.55)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 0.45 (5.39)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 1.40
Total Estimated Cost to Develop                           = $ 84,939
 (average salary = $56,286/year, overhead = 2.40).
SLOCCount, Copyright (C) 2001-2004 David A. Wheeler
SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL.
SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to
redistribute it under certain conditions as specified by the GNU GPL license;
see the documentation for details.
Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."

Test case with Linux-kernel.git

Totals grouped by language (dominant language first):
ansic:     16759585 (97.89%)
asm:         272127 (1.59%)
sh:           29511 (0.17%)
perl:         27344 (0.16%)
python:       17862 (0.10%)
cpp:           5063 (0.03%)
yacc:          4648 (0.03%)
lex:           2583 (0.02%)
awk:           1385 (0.01%)
ruby:            25 (0.00%)
sed:              5 (0.00%)

Total Physical Source Lines of Code (SLOC)                = 17,120,138
Development Effort Estimate, Person-Years (Person-Months) = 5,574.59 (66,895.03)
 (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 14.20 (170.45)
 (Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 392.47
Total Estimated Cost to Develop                           = $ 753,050,754
 (average salary = $56,286/year, overhead = 2.40).
SLOCCount, Copyright (C) 2001-2004 David A. Wheeler
SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL.
SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to
redistribute it under certain conditions as specified by the GNU GPL license;
see the documentation for details.
Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."

``

Applied


"Allow Always" does not appear to be working

#3720 and #3612 don't to seem have mention of an additional issue.
We use an external service for self monetization and also to use an API for one of our features.
Choosing "Allow Always" when prompted by “The extension is making a request to an external service. Do you want to allow this request" does not appear to stick. In some cases it's necessary to accept multiple times, and then the calls fail anyways (return a 500, never hitting the external service).
The only way to get it to work is to choose "Allow Once," which is causing the user to have to accept many times. Or they choose "Allow Always" and it looks like our app isn't working.
Will the "Allow Always" feature work soon? Is there an idea of when? If not is there a workaround so we can avoid this dialogue and having the setting block our features?

Hi @strophanthus ! The option does not stick due to the licensing issue mentioned in one of the bugs. This prompt only happens in a sandbox environment which is meant for testing. Can you try updating your license? After the license update, inserting a record into the NAV App Setting table should work without raising any permission errors.

Hello. I updated the VM and can now see the "Configure" option, enabling HTTP calls from that screen works. Thanks!

Today, I updated my container with new docker container and facing same issue.
current version :

message which i recived when we try to use our app :

I understand that for this error you require an updated license.
The "enable HTTP" option is in the Extension Management page. If you click the ... on your app's card, and go to Customize, there is a toggle there.

As of now Configure option is not present under Extension page, so used this blog and updated Runtime in app file as '1.0'
Blog : https://docs.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/devenv-developing-for-multiple-platform-versions

@chavansachin20 have you updated your license?

@atoader , I'm using NAV 2018 dev license which I've used for App v1 & V2 development .

Configure option visible only when you use latest License.
follow below steps:

Generate new license
Create new container : use same license while creating a docker container
Open NAV client > search for extension management > select your app > Click on manage > select Configure > tick "Allow HttpClient Requests"


Settings subcommand

rebase and add Steps into settings action

Reviewed settings changes. +1 on @thedoubl3j 's comment.


sync data at the k8s

Question
when i deploy shenyu-admin and shenyu-bootstrap in the k8s,
example :
shenyu-admin has three pod
shenyu-boostrap has four pod
they can sync data correctly?
@moremind @yunlongn

Please pay attention to the port communication


Add option to output check results in a tabular format

What does this PR do?
This adds a new --table option to the agent check subcommand, allowing for the Series data to be formatted in a more condensed table.  It also sorts the metrics by name, and then by tags to help organize the output.
Example:
root@docker-desktop:/# agent check scylla --table
=== Series ===
  NAME                                                 TIMESTAMP   VALUE  TAGS
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  110    [server:localhost:9180 shard:0 type:histogram upper_bound:1024.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  0      [server:localhost:9180 shard:0 type:histogram upper_bound:16.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  134    [server:localhost:9180 shard:0 type:histogram upper_bound:262144.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  134    [server:localhost:9180 shard:0 type:histogram upper_bound:32768.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  7      [server:localhost:9180 shard:1 type:histogram upper_bound:1048576.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  7      [server:localhost:9180 shard:1 type:histogram upper_bound:131072.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  7      [server:localhost:9180 shard:1 type:histogram upper_bound:262144.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  7      [server:localhost:9180 shard:1 type:histogram upper_bound:4096.0]
  scylla.storage.proxy.coordinator_read_latency.count<PHONE_NUMBER>  7      [server:localhost:9180 shard:1 type:histogram upper_bound:65536.0]

Motivation
Large metric output in JSON format becomes quite difficult to parse and grep easily without post processing scripts.  Formatting in a table can make the data easier to absorb quickly.
Additional Notes

The sorting by tags is best effort as can be seen in the example above.  It will group appropriately by shard but the upper_bound tags are interpreted as strings and thus out of order.

This adds a new external dependency, which seems relatively popular.  Of course, this could be also be coded manually to avoid the dependency if needed.

This PR only addresses Series output, the other check output values such as Service Checks remain unchanged.

Awesome! WDYTA putting the timestamp at the end since we wouldn't need it usually?

@ofek The only issue with that would be the alignment of the columns.  Since tags are so variable they can extend the line considerably, and having timestamps at the end would muddy the output and make it harder to see things at a glance.


Use domainName instead of userDomainName when Token with User Name authentication type is selected

When creating an OpenStack provider with Token with User Name authentication type, the domainName/domainId is needed in the secret, while the userDomainName is not a must.
Currently, there is no Domain Name filed on creating provider page, the provider will be in connectonfailed status. Need to change the User Domain Name field to Domain Name

@qiyuann @mmartinv hi,
the Jira ticket say to use userDomainName:
https://issues.redhat.com/browse/MTV-542?focusedId=22280243&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-22280243
do you have a newer version of the fields to use ?

merged: https://github.com/kubev2v/forklift-console-plugin/pull/596
@qiyuann please close after verification

User Domain Name is changed to Domain Name now, the provider could reach to Ready status.


Tutorial: Rate Limiting a service with Kong Gateway

Preview URL: https://deploy-preview-43--kongdeveloper.netlify.app/tutorials/add-rate-limiting-to-a-service-with-kong-gateway/

Revised the existing RL service tutorial to clean up the prereqs
Made some metadata for prereqs, it doesn't work and I'm not sure how to fix it, but hopefully that gives a starting point for what we could do
Added links to relevant pages to create a better user journey to find this content

My testing experience notes:
Disclaimer: I've used decK like once, so I'm very new.

Used the quickstart to quickly get Kong, no problems. Already had Docker and curl.
I needed a service and a route. I used the rendered examples here (since they aren't rendering in my prereqs because I've got the formatting wrong).

These aren't commands, what do I do with these? Looking at https://docs.konghq.com/deck/1.38.x/guides/getting-started/#export-the-configuration I realized these are part of the kong.yaml config. I feel like it was too hard to run these on their own separately, so I just copied them into my kong.yaml file.
Note: I did fix the weird - formatting when I copy/pasted that Lucie reported.

While I was at it, I also copied in the enabling the RL plugin into the yaml file just so it was easier.
Okay, I have my file edited, now how do I push my changes? Found this section: https://docs.konghq.com/deck/1.38.x/guides/getting-started/#diff-and-sync-the-configuration-to-kong-gateway

diff worked just fine
ran into errors with the host when I ran sync

Summary of errors and what I fixed to make it work:

services needed a host, or so Terminal told me. I added a host before the service name.
The prereqs made me think that the routes needed to be on a separate line from services, when really, what worked for me was making routes part of the services, for example:

_format_version: "3.0"
services:
  - host: example.com
    name: example_service
    routes:
      - name: example_route

I also continued to get errors until I added a path to the route. I used this to help me figure out the right routes formatting: https://docs.konghq.com/deck/1.38.x/guides/defaults/#test-default-value-handling

What I originally had for the formatting and thought it had to be (which didn't work):
_format_version: "3.0"
services:
  - name: example_service
routes:
  - name: example_route
    service:
      name: example_service
plugins:
  - name: rate-limiting
    service: example_service
    config:
      second: 5
      hour: 1000
      policy: local

However, how will this work when we aren't only using decK? I don't know if it's safe to assume people know to run those. I'd like to mention that the config goes in a kong.yaml file, but again, that would prevent the step instructions from being reused for Admin API, Konnect UI, etc.

We can assume that tutorials are deck only for now. We may revisit in the future, but it's safe to add deck specific instructions

cherry-picked changes in https://github.com/Kong/developer.konghq.com/commit/620320ed86287be34c8a556ad0bcc31ed47ae6fc